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ABSTRACT 
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and students. As the demand from network users grows rapidly, many 
network administrators are asking for resources to meet that demand; 
their requests are being met with questions about the costs, uses, 
and effects of networking at their institutions. In most cases, the 
information required to answer those questions is not available, nor 
are the tools with which to collect that information. This 
investigation attempts to characterize the current state of 
networking in academic institutions and to identify and develop tools 
and approaches to assess academic networking. Data was collected 
primarily through focus groups, interviews with individuals and small 
groups, and site visits to selected academic institutions. The 
primary method of data analysis was content analysis. Results of the 
study indicate: (1) there are multiple barriers to the assessment of 
academic networking; (2) there is little current assessment of 
academic networking; (3) there is an interest in assessment of 
academic networking and a need for assessment tools and techniques; 
and (4) among the tools and techniques desired are guidelines for 
conducting qualitative assessments, a set of quantitative measures, 
and a user survey instrument, (Contains 94 references.) 
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1. INTRODUCTION 



This document is a final report of the research project. Assessing the Academic Networked 
Environment, funded by a Research and Demonstration Grant from the U.S. Department of 
Education, College Library Technology and Cooperation Grants Program. This research project 
was conducted by a team of faculty and graduate students at Syracuse University’s School of 
Information Studies. i The study began in October, 1994, and was completed in April, 1996. 
Background to the research as well as the methodology and findings are reported here. 

Another product of this research is an assessment manual. 2 The manual is a collection of 
strategies and options for assessing networking which were developed as part of this investigation. 
Included in the manual are qualitative assessment techniques, quantitative performance measures, 
and a survey instrument for obtaining network users’ perspectives. 

1.1. Purpose of the Research 

The provision of network access, resources, and services to members of the academic 
community is becoming an area of utmost interest and importance to academic administrators, 
faculty, and students. As the demand from network users grows rapidly, many network providers 
are struggling to meet that demand and their requests for resources to do so are being met with 
questions about the costs, uses, and effects of networking at their institutions. In most cases, the 
information required to answer those questions is not available nor are the tools to collect that 
information. It is within this context that this investigation was conceived. 

The purposes of this research were: to define and describe the academic networked 
environment and its component parts; to identify existing measures of network performance and 
impacts; to identify factors which facilitate or inhibit network assessment; and to develop and 
operationalize additional measures. Following are the research questions which guided this 
investigation: 
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• What information technologies and services comprise networked information, and 
to what degree are these similar across various academic institutions? 

• Who are the "users" of networked information within the academic setting and how 
might we develop a typology of such users? 

• What are the organizational structures used in academic institutions to provide 
networked information services? 

• What are the key factors that appear to affect the overall success of the networked 
environment in an academic setting? 

• What measures can be developed to assess the impacts of networking on the 
academic environment? 

These questions formed the foundation on which this investigation was conducted to characterize 
the current state of networking in academic institutions and to identify and develop tools and 
approaches to assess academic networking. 

1.2. Overview of the Report 

Included in this report are: a review of relevant research and information resources which 
constituted the background that informed this investigation, a review of the methodologies 
employed in this investigation, and the results of this investigation. 

Because the assessment of academic networking has not been the focus of much previous 
research, the relevant published literature is minimal. For this reason, the background 
investigation went beyond the published literature to include a review of existing sources of data on 
computer use at academic institutions and an analysis of information technology (IT) plans 
produced by a selected group of academic institutions. 

The focus of this investigation was on academic institutions which meet the criteria for 
Carnegie Research I and II institutions and which are also members of the Association of Research 
Libraries (ARL). The methodology used in this investigation weis an evolving one, combining a 
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variety of qualitative techniques for data collection and analysis in an exploratory approach. Data 
were collected primarily through focus groups, interviews with individuals and small groups, and 
site visits to selected academic institutions. The primary method of data analysis was content 
analysis. 

Results of this study indicate the following: 

• There are multiple barriers to the assessment of academic networking. 

• There is little current assessment of academic networking. 

• There is an interest in assessment of academic networking and a need for 
assessment tools and techniques. 

• Among the tools and techniques desired are: guidelines for conducting qualitative 
assessments, a set of quantitative measures, and a user survey instrument. 

Thus the primary contribution of this study is a framework for academic network 
assessment which is suitable for field testing by academic institutions. This framework is not a 
guide nor does it encompass step-by-step procedures in a start-to-finish approach to assessment. 
However, it does provide a stealing point for institutions which are interested in assessment. The 
tools, techniques, and measures developed during this investigation are by no means presented as 
appropriate for all institutions. Rather they are presented as a menu of items from which 
instimtions can select. The study team which conducted this investigation anticipates that 
individual institutions will begin implementing and further developing these tools, techniques, and 
measures in order to determine which of them work best in certain kinds of institutions. The study 
team further expects that these tools, techniques, and measures will evolve over time as they are 
field-tested and fine-tuned by institutions. 
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Notes to Section 1 

1. Co-principle investigators for the study were: Cynthia L. Lopata, Assistant Professor; and 
Charles R. McClure, Distinguished Professor. The following graduate students also 
contributed to this investigation: Bill Boroson, Anne Diekema, Kristin Eschenf elder. Bill 
Gibbons, Denise Masters, Makiko Miwa, Diane Sotak, Claire Urfels, and Jean Van Doren. 

2. This manual. Assessing the Academic Networked Environment Strategies and Options, is 

available from the Coalition for Networked Information (CNI), in Washington, DC. For order 
information email pubs@cni.org or call 202-296-5098. 
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2. BACKGROUND 



2.1. Introduction 

The assessment of networking in academic institutions is a topic which has not been well 
researched to date. In fact, assessment of networking in general, and more broadly of all computer 
technologies, to determine their impacts on the organizations which have adopted them is an area 
about which little is known. Thus there was not a great deal of previous research to form a 
foimdation on which to build this investigation. The study team consulted a variety of sources, in 
addition to the published research literature, to identify and collect information which might inform 
this study. Results of the background investigation which preceded, and formed the foundation 
of, this study are presented here in four parts: a review of relevant literature, a review of existing 
network models, a review of existing sources of relevant data, and an analysis of existing plans for 
IT development and support at academic institutions. 

2.2. Literature Review 

This section reviews the literature on measurement and evaluation of computer-based 
information systems and network-based technologies, resources, and services. Although the 
specific context of interest for the study is the academic setting, the review also considered other 
contexts, e.g., business, in the hope of identifying studies that had the potential of being useful in 
an academic setting. This review covers literature from the fields of education, 
business/management, library/information science, and computer science. The review focuses 
primarily on literature published within the last ten years and, rather than being exhaustive, 
highlights selected studies and reports which are representative of certain segments of the broader 
literature. The objectives of the review were to: 

• Assess the degree to which successful measurement and evaluation of networked 
and computer-based services occurs. 
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• Understand better the barriers that inhibit successful measurement and evaluation of 
networked and computer-based services. 

• Identify approaches that may be useful for developing measures to assess the 
impacts of networking in an academic setting. 

There is much support in the literature for the measurement and evaluation of IT and 
networking in academic institutions. According to Heterick ( 1994), "Our colleagues want such 
measures, our legislatures and funding agencies are demanding such measurements, and the future 
health and well-being of our profession require such measurements" (p. 64). Heit (1994) says 
that, "The combination of greater scrutiny brought on by both increased visibility and a growing 
demand for accountability make this an excellent time to do an assessment of the institution's 
information technology" (p. 3). 

In spite of this generally recognized need for assessment, the literature reveals few attempts 
to develop applicable measures or to conduct assessments of IT and networking in academic 
institutions. McClure (1991) gives several reasons for the lack of research in this area. First, the 
evolution of the academic network has been incredibly fast, leaving the participants with barely 
enough time to stay abreast of new technologies let alone time to study the developments. Second, 
academic environments are typically slow in providing evaluations and a “culture” erf" ongoing 
evaluation is virtually nonexistent. And third, the development of useful, feasible, and meaningful 
measures is extremely difficult 

In reviewing the relevant literature, the study team identified foiur major categories of 
research: program and services evaluation, IT measurements in business, IT measurements in 
education, and IT measurements in libraries. Each of these categories is reviewed in a separate 
section below. 

2.2.1. Definitions 

A number of terms are used repeatedly throughout the various literatures reviewed here. 
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These often-used terms are defined below. 

Measurement is the process by which numbers are assigned to describe some object or 
phenomenon in a standardized manner. As Kaplan ( 1964) writes, it “is a device for 
standardization, by which we are assured of equivalences among objectives of diverse origin”(pp. 
173-174). Measurement may or may not lead to evaluation; evaluation often requires 
measurement. 

Evaluation is the process of identifying and collecting data about specific services or 
activities, establishing criteria to assess their success, and determining the degree to which the 
service or activity accomplishes stated objectives. As such, evaluation is a decision making tool 
intended primarily to assist decision makers to allocate resources that best accomplish 
organizational goals. Evaluation reflects value judgements on the part of the evaluator regarding 
the adequacy, appropriateness, and success of a particular service or activity. 

Performance measures represent a broad managerial concept that encompasses 
measurement of both inputs (indicators of the resources essential to provide a service) and outputs 
(indicators of the services resulting from the use of those resources). The primary purpose of a 
performance measure is to serve as a self-diagnostic for the organization. Simply stated, 
performance measures ask decision makers to answer the question: How well is the service or 
activity doing what it claims to be doing? 

Information technology is described by Emery (1987) as "the technology associated with 
the computer hardware, computer software, and communications" (p. 317). Tanenbaum ( 1981) 
described a computer network as an "interconnected collection of autonomous computers. Two 
computers are said to be interconnected if they are capable of exchanging information. The 
connection need not be via a copper wire; lasers, microwaves, and earth satellites can also be used" 
(p. 2). According to Boyce, Meadow, and Kraft ( 1994), “An information system is a set of 
components whose combined purpose is to acquire, process, and transmit information” ( p. 5). 
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2.2.2. Program and Services Evaluation 

The evaluation of programs and services is one area of the relevant literature where a 
significant amount of research has been done. Much of this research applies to libraries, although 
some of it applies to other types of organizations, including computing centers and information 
systems departments. 

Library administrators and researchers alike have struggled for years to measure and 
evaluate the different programs and services provided by their libraries. Traditionally the 
measurement of library services has mainly focused on output measures but it has recently evolved 
into more meaningful measures like impact measures. The literature on measuring library services 
is quite extensive (e.g.. In Service to Iowa, 1989; Library Programs, 1990; Lynch, 1983; 

McClure, 1986; Murphy, 1987; Owen, 1991; and Shaughnessy, 1987). 

King and Griffiths (1991) established four general categories of measures for use in 
libraries and information centers: size, goodness, use, and usefulness. Within these general 
categories, they developed a variety of more specific measures, including: 

• Input measures including amount of resources (e.g., staff, equipment and systems, 
facilities, collection, etc.) applied to operational functions and services, amount of 
funds/money applied and relevant attributes of resources applied (e.g. staff 
competencies, equipment reliability, etc.); 

• Output measures including quantities and attributes of output such as quality, 
timeliness, availability and accessibility; 

• Effectiveness measures including amount of use, user's perception of output 
attributes, user-expressed satisfaction with services and specific attributes, user- 
indicated importance of services and attributes, purposes of use and consequences 
of use of services. 

• Service domain measures including total population size and attributes, user 
population size and attributes, size and attributes of geographic area and information 
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needs and requirements. 

According to King and Griffiths (1991), combinations of measures which represent the 
relationships among them are even more meaningful than individual measures. They developed the 
following indicators, based on such combinations: 

• Operational performance indicators which relate input and output measures and their 
attributes (e.g. productivity-output divided by input, cost per transaction at various 
levels of quality, etc.); 

• Effectiveness indicators which relate output quantities and attributes to use (e.g. 
extent of use as a function of quality and timeliness of services); 

• Cost-effectiveness indicators which relate input cost measures, effectiveness 
measures and domain measures (e.g. cost per use, cost peruser, cost per capita); 

• Impact indicators which relate actual use or needs to potential use or needs (e.g. 
users per capita, use per capita, needs fill rates, etc.). 

King and Griffiths ( 1991) also investigated the economic value of library services by 
examining: 

• What users are "willing to pay" (i.e., as demonstrated by what they actually do pay) 
for information provided by special libraries, in terms of their own time to locate, 
obtain and use the information and monies paid; 

• The difference in the cost of using library services and what it would cost to use 
alternatives, if the services were not available; and 

• The "savings" obtained from reading library-provided materials that would likely be 
lost if the services were not available (based on the assumption that the total 
"willingness to pay" as defined above would not change). 

One library service that has been studied intensively is reference. There have been two 
major approaches in evaluating reference: quantitative measurement and qualitative measurement 
Gorman's (1987) book is based on two surveys, one for quantitative and one on qualitative 
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measures. Quantitative measures focus on the number of reference interactions completed, the time 
to complete them, etc. The object of study in measuring the quality of reference services is the 
reference question outcome. Hemon (1987) and Bunge (1990) discuss measuring success rates in 
answering questions rather than just gathering reference statistics. Hemon and McClure (1987) 
suggest that reference services can be measured unobtrusively. Murfin and Gugelchuk (1987) 
describe an instmment that judges the success of a "reference outcome" based on what the patron 
wanted to find. Green (1988) stresses the continuous measuring of reference service 
effectiveness. 

Beyond the measures described above, which relate to specific programs and services, 
other measures of institutional performance also encompass the evaluation of such programs and 
services. Orr's schema as described by Buckland (1992) may be used for measuring the quality 
and value of libraries at the institutional level. Buckland concludes that each concept is quite 
different from the other and both are difficult to measure. Other research on measures at the 
institutional level includes; McClure et al. (1992), who addressed the creation of an international 
standard for library p)erformance measures; Shaughnessy (1990), who wrote about library 
effectiveness; and Afshar (1991), who provided a theoretical basis for measuring the quality of 
institutions of higher education. 

Program and service evaluation are also encompassed in institutional assessment measures 
which were develop>ed for academic computing organizations. Heit ( 1994) provides a number of 
close-ended questions on topics such as planning, policies and procedures, facilities and staff, 
products and services, organization and external relationships, and funding. Computing centers 
can use these questions to do self-evaluations. Rubin (1991a, p. 77, 78) provides ten possible 
metrics to gauge the p>erformance of the information systems department: 

1 . 




2 . 



Productivity metrics . These measure the software delivery rate and ability to 
suppxjit software. 

Quality metrics . These measure the technical quality of the software produced and 
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maintained, the software's functional quality in the context of meeting business 
needs and the quality of the software engineering process. 

3 . Delivery metrics . These measure the organization's ability to meet time and cost 
commitments. 

4. Penetration metrics . These measure the extent to which tools and techniques have 
been successfully disseminated. 

5. Work profile metrics . These measure the effort and elapsed time it takes for work to 
progress through life cycle stages. 

6. Demand metrics . These measure request backlogs and the organization’s ability to 
service them. 

7. Technology assimilation metrics . These measure the organization's ability to adopt 
and assimilate promising new software engineering technology. 

8 . Work distribution metrics . These measure the balance between maintenance and 
development. 

9. Capability metrics . These measure the ability of the organization to manage, 
measure and improve itself. 

10. Business-oriented metrics . These link IS functions to the success measures used by 
the business to gauge business performance. 

2.2.3. IT Measurements in Business 

Carlson and McNurlin (1992a) observe that trends such as total quality management and 
business process reengineering have focused the business community’s interests on evaluating the 
contribution of IT to their organizations’ success. The management literature on the evaluation of 
IT is extensive and can be subdivided into: general information system measures, value of 
information systems, information system performance, quality and effectiveness of information 
systems, productivity of information systems, and measurement of end-user satisfaction. 
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According to Rubin (1991a, 1991b) effective information systems measurement programs 
involve both information systems measures and business measures. In order to decide which 
measures to use, an organization has to complete several steps (Rubin 1991c, p. 79): 

• Identif y all audiences f or measurement and measurement stakeholders i 

• Analyze the measurement needs of each audience and stakeholder2 

• Produce a map that cross-references audiences to needs 

• Produce a map that cross-references needs to possible metrics 

• Decide which candidate metrics to use 

• Establish priorities and a phased implementation plan. 

These steps will result in two matrices that provide organizations with an overview of all necessary 
measurements. This solves the problem, identified by Carlson and McNurlin (1992b), of deciding 
which performance measure to use in a given situation. 

In the literature on the value of information systems the importance of measurement is 
discussed, but few useful measures have been proposed. Ryder (1988) and Sullivan-Trainor 
(1989) stress the need for measuring the business value of IT or facing budget cuts. Lane and Hall 
(1989, pp. 73-74) provide six general principles that should form the groundwork for the actual 
measurement rules but do not suggest exactly how the value should be measured. These are: 

• Recognize information systems as an asset-based function 

• Agree on how to quantify intangible value 

• Value existing technology as a point of departure 

• Establish a procedure to "investment justify" systems projects 

• Review and update the valuation on a regular basis 

• Integrate the systems-valuation process into business planning 

A welcome exception is Ranter’s (1987) description of the value chain evaluation method 
which compares the cost of producing a service or product to the price customers are willing to 
pay. Campbell (1992a) explains that in banking the value of IT can be assessed by measuring the 
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effect on the four income driverss since these comprise the bulk of a bank's net income. 

The literature on information systems performance discusses measures like the Real-Time 
Kernel (RTK)4 performance (Berggren, Gustafsson, and Lindh, 1992). This study suggests that 
measuring the performance of the RTK can be useful to establish the usage of a RTK or to 
compare different RTK's. Baccelli and Fayolle (1983) combine computer science with probability 
analyses to evaluate system performance. Some authors like Heterick (1994) have been 
questioning these traditional system performance measures while others suggest using them in 
combination with other measures (Lane and Hall, 1989). 

Slevin, Stieman, and Boone (1991) discuss the critical success factor (CSFO process as a 
measurement of performance. Instead of using the key areas of an organization for plaiming 
purposes, they can be used to design performance measures. In this case study, conducted in a 
university environment, the authors tried to define specific critical success factors for each 
department and determine how each could be measured. The CSF method enables "measuring the 
performance of a technologically sophisticated set of services provided to a variety of users" (p. 
170). 

Evans, et al. (1988) describe four methodologies to evaluate a system's effectiveness; 

• The IDEF methodology for tmderstanding an environment, its functions, 
relationships, and information flow 

• The Bailey and Federle Methodology for understanding user needs and potential for 
computer support 

• The Krobock Methodology for determining utility of information systems 

• The Pearson Methodology for determining user satisfaction with information 
systems (p. 235). 

The IDEF methodology models information flows and functions in an enviromnent in order 
to make the workings of an organization more clear. In the Bailey and Federle methodology users 
identify their key tasks, rank them according to imp>ortance and time spent on them. Using this 
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methodology provides insight into users’ work and their information needs. The Krobock 
methodology "measures multiple factors which affect information usefulness including timeliness, 
currency, accuracy, relevancy, adequacy, useability, and handiness" (Evans et al. 1988, p. 227). 
The users themselves decide on these factors. The above methods can be used individually or in 
combination with each other. The Pearson Methodology "uses a semantic differential technique to 
rate 39 factors which cause satisfaction or dissatisfaction with information systems" (p. 228). 

Although the quality of an information system is often intangible, there have been some 
attempts to develop measures of this quality. Rivard and Kaiser (1989) suggest using focus 
groups, in-depth interviews, expert opinion, the Delphi technique, observation, benefit profile 
charts, and existing systems analysis methods to help measure quality of information systems. 

A series of studies investigating the relationship between IT and organizational productivity 
describes this phenomenon and refers to it as the productivity paradox (Brynjolfsson, 1993). This 
research suggests that even though companies may invest a lot in IT, this may not increase 
productivity. Much of the literature on the productivity of information systems focuses on this idea 
of the productivity paradox (e.g., Dubashi & McGough, 1992; Due, 1993; Freeman, 1989; Krohe, 
1993; Laberis, 1993; Lane & Hall, 1989; Magnet, 1994; McGough, 1992; McGovern, 1993; 
Shaw, 1994; Strassman, 1985, 1990). 

This productivity paradox might be due, in part, to the use of inappropriate measures. 
Researchers used traditional productivity measurements^ that, according to Brynjolfsson ( 1993) 
"fail to account for nontraditional sources of value" (p. 73). Values like quality increase and 
increased customer satisfaction due to better and faster service are intangible and difficult to 
measure. Some studies also failed to take time into account It can take several years for benefits 
to appear in the form of improved productivity. It could also be that using IT might be profitable 
only to some companies or that we need to learn how to manage IT before it will become profitable 
(Brynjolfsson). 

Krohe (1993) explains that investing in IT does not always pay off because computers can 
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do things faster but not necessarily better. The extra time gained by using computers might be lost 
once people have learned how to use them and after computer problems have been solved. 
According to Krohe the average PC user wastes 5. 1 hours a week "futzing" with the computer. 
This phenomenon, also called the electronic sandbox, is described as a virtual place where people 
go to play computer games and chat. 

Koenig and Wilson (1996) give several additional reasons for the productivity paradox. 
First, there seem to be some benefits of IT that are simply not captured by any productivity 
measures. For example, "What is the utility of higher definition television, or of a vastly increased 
number of channels?" (Koenig and Wilson, 1996, p. 2). Second, there are benefits that could be 
measured but are not. One example of this type of benefit is increased customer satisfaction. 
Online information resources such as Chemical Abstracts or Beilstein may not have affected sales 
in the pharmaceutical industry, however, the pharmaceuticals being produced might be more 
effective. Third, the benefits of IT are often lost in maintaining a competitive advantage. Using IT 
as a product differentiation enabler might spoil productivity gains. And lastly, Koenig and Wilson 
mention that the productivity measures might not be useful in a service industry. 

As shown in the previous paragraphs, benefits from IT can come to an organization in 
ways other than those which productivity measures tend to capture. As Krohe (1993) said, 
“Applying conventional economic indexes to computer productivity... is like measuring air 
pressure with a ruler” (p. 19). 

Lee (1988) suggests that the classic productivity measures of output and input pose 
problems in the case of IT since the outputs are so difficult to identify and measure. This view is 
shared by: Allen, 1986; Duncan, 1992; and Rudd, 1993,who blames the nature of information for 
the productivity measuring difficulties. Belitsos ( 1988) rightly points out that the traditional input 
and output measures will remain in use until new ones are accepted. 

Y et some contend that the productivity of information systems is meiisurable. Oman and 
Ayers (1988) distinguish five aspects of productivity to which a cost benefit analysis can be 
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applied: 



• Document output productivity 

• Secretarial/clerical productivity 

• Professional/managerial productivity 

• Office productivity 

• Organization productivity 

Document output productivity "is determined by comparing the volume of documents or 
documentation produced per time unit on conventional equipment to the volume of documentation 
produced using automated information technology" (Oman and Ayers, 1988, p. 33). Measuring 
the secretarial/clerical productivity involves measuring quality of service, the rate of persoimel 
turnover, morale and status of secretaiial/clericad functions. Professional/managerial productivity 
can be measured by looking at "the effect of information technology in saving professional time on 
document creation and revision and on the ability of information systems to improve the access to 
and quality of information" (p. 34). Office productivity "is measured by the contribution of the 
office system to the efficiency and effectiveness of the overall organization" (p. 34). Orgamzation 
productivity explores "the effect of information technology on productivity by examining the 
impact on achieving organization goals" (p. 34). Oman and Ayers further provide three benefit-cost 
models that can be used on one or more of the five productivity approaches. The first is the cost- 
reduction model that is based on the assumption that automation will lead to reduced labor costs. 
The second model examines the "benefits of information technology while assuming the costs stay 
the saune" (p. 36). The third is a mixed costs/benefits model. 

Tayntor (1994) suggests combining metrics and productivity programs which means 
quantifying services and demonstrating productivity. Scudder and Kucic (1991) recommend the 
use of multiple measures dealing with efficiency as well as effectiveness. A list of "overall 
performance measures" (p. 350) describes measures concerning persoimel performance (e.g., 
business knowledge, technical capabilities), managerial performance (e.g., performance audits. 
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attitudes of users), developmental performance (e.g., staff turnover, documentation quality), goal 
setting (e.g., quality of planning, forecast of future IS capabilities), operational performance (e.g., 
user interaction, timely output), and financial performance (e.g., budget performance, cost 
recovery). 

Another productivity measure is the function point analysis. As described by Bock and 
Klepper (1992), it is used to measure the work output of information systems. Originally 
developed by Albrecht, the function point analysis counts and projects deliverables of an 
application by looking at the number of inputs to the application, the number of outputs from the 
application, the number of inquiries an end-user can make, the number of logical data files used by 
the system, and the number of interfaces to other publications. Function points are calculated for 
each deliverable. Jebber ( 1992) and Laplante ( 1991 ) both describe computer packages that 
measure productivity. The measures used by these packages range from the number of lines of 
code to industry benchmarks to compare and contrast how an organization is doing. 

Two aspects of this paradox may be considered as they relate to the academic environment 

• The culture of the academic institution, in contrast to the for-profit, business 
environment, may not inherently possess a strong inclination to maximize 
efficiencies in the deployment of information technologies. Such a scenario would 
be analogous to Roach's ( 1991) assessment of information technology in the 
service sector in which he suggests that economic protection, afforded this sector, 
contributed to expenditures without sufficient regard to efficiencies, resulting in 
negative productivity gains. 

• A 1994 study, commissioned by the Computer Science and T elecommunications 
Board (CSTB) of the National Research Council, concluded that, for service 
organizations, many of the macroeconomic measures of productivity mask or 
overlook critical performance assessment criteria which may be derived through a 
more qualitative methodology. Of particular significance for academic institutions 
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would be those performance measures associated with the overall quality of service 
provided to their constituents (CSTB, 1994). In fact, many participants in this 
study were led to conclude that "the whole concept of productivity, as it is currently 
measured, may be outmoded and that other measures of performance provide a 
better indication of the contribution of information technologies" (CSTB, p. 29). 

Doll and Torkzadeh ( 1991) measure user computer satisfaction using Likert scales, which 
they believe are very useful tools in discovering a respondent’s beliefs. Arkush and Stanton 
(1988) measure the value of end-user computing by calculating a sample value (sample benefits 
minus sample costs) instead of a total value. Although calculating a total value is more precise, 
they reason that using only a sample of the total user population and extrapolating the results is a 
much more cost-effective way to measure. 

Some authors suggest evaluating the performance of the information systems department 
(not just the system) by asking users about the department's performance. In a roundtable 
discussion documented by Rhodes and Chester ( 1987) the users' role in evaluation is also 
mentioned. 

2.2.4. IT Measurements in Education 

Henry et al. (1993) discuss the possible effects of networks on the way information is 
being transferred. Communication between disciplines is easier and more projects will be 
completed collaboratively. Also the scholarly publication process is likely to change. Rickman 
and Hubbard (1992) provide a description of the features of the electronic campus system of 
Northwest Missouri State University (Maryville). 

Anderson and Draper ( 1991) show the wide range of areas involved in the evaluation of 
learning. Evaluation of learning involves research on mental representation, learning mechanisms, 
motivation, teaching, didactics, teaching methods and media, social interactions and instimtional 
factors. Anderson and Draper argue that IT influences all of these areas thus making the 
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evaluation of the impact of IT on learning an extremely complex matter. Anderson and Draper also 
suggest that "educational benefit is not at the level of the medium, but at some finer grained level 
concerned with how the medium is used” (p. 5). 

Goodhue et al. (1993) use a wide scope survey, selected in-depth interviews, and usage 
statistics to measure the impact of the Integrated Information Center (ICC) at the university of 
Minnesota. In doing so Goodhue et al. try to measure how technology affects performance. The 
two premises on which this study is based are: technology must be a good fit for the task, and 
technology must be used. Only if these two premises are satisfied can performance increase. 
However, from the article it is unclear as to how the technology fit and the use of technology are 
measured. 

A study using similar data collection methods was done by Wilson et al. (1993) who 
evaluate the impact of technology at Peakview Elementary School by comparing their networked 
school to other schools. This study looked at student achievement, the use of technology, and the 
impact of technology on teaching. Data were collected at the beginning of the school year and at 
the end of the year in order to identify changes. Data were gathered from students and teachers 
using written surveys and interviews. Among the things that they looked at were student 
achievement, use of technology, and impact on teaching. For data interpretation the researchers 
used bar charts and line charts. They also calculated the chi square or analysis of variance where 
possible. 

Gearhart et al. (1990) describe a program that measures the impact of educational 
technology by "observing" the classroom at timed intervals. "Snapshots" of classroom activity are 
created by coding a large number of class routines regularly. These descriptions can be used to 
report what happens in the classroom and also identify where technology has an impact on teaching 
and learning. The authors agree that this instrument should be used in combination with other data 
collection methods in order to be useful. Matthews and Winsauer ( 1984) offer criteria to evaluate 
educational software by using concepts from learning theory. 
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Reit (1994) is an exception to the literature since she provides a number of clear close- 
ended questions for academic computing centers to determine how they are providing various 
services. Many negative answers indicate that the center is in bad shape. The questions are 
divided among several topics: planning, policies and procedures, facilities and staff, products and 
services, organization and external relationships, and funding. One weakness of this method is 
that the opinions of users may be inaccurate and represent only their “perceptions” of the services 
being provided. Nonetheless, such information can be very valuable. 

2.2.5. IT Measurements in Libraries 

Lynch (1988) discusses the measurement of system performance based on MELVYL, the 
online library catalog of the University of California. Lynch recognizes three types of data to be 
collected: statistical data, benchmark data, and detailed logging data. Performance measures are 
useful for capacity planning, acceptance testing, quality assurance, and system tuning and 
refinement. Measuring response time is especially useful in determining user satisfaction. He 
believes that measures like number of searches, number of displayed records, and number of help 
screens viewed are of less importance. 

Peters (1988) and Schmidt et al. (1988) discuss the possible standardization of 
performance evaluation of automated library systems. Peters proposes a framework for the 
development of performance testing of library automation systems. Regazzi ( 1988) evaluates an 
information retrieval system by doing an experiment using human subjects to establish relevance of 
certain documents in relation to search topics. Using relevance, recall, and precision as 
information retrieval system measures is a much debated subject in ttie literature (Boyce, Meadow 
& Kraft, 1994; Eisenberg & Barry, 1988; Froehlich, 1991; Harter, 1992; Janes & McKinney, 
1992; Park, 1993; Sal ton, 1992; Schamber & Eisenberg, 1991; and Schamber, Eisenberg, & 
Nilan, 1990). Such approaches, however, require sophisticated methodologies that would be 
difficult for practitioners to use. 
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2. 2. 6. Conclusion 

Overall, the review of the literature suggests that there have been few research efforts to 
develop and test measures of the impacts of networking in an academic setting. However, there is 
much agreement that such measures are important and should be developed. Following is a 
discussion of some key concepts which have emerged from this literature and which form a 
foundation for the development of measures relevant to this investigation. 

Traditional performance measures can be divided into four assessment indicators: 
extensiveness, efficiency, effectiveness, and impact (King and Griffiths, 1991; McClure, 1991; 
Menou, 1994). McClure (p. 37) describes these measures as follows: 

• Extensiveness: how much of something has been provided, e.g., number of 
transactions on a listserv 

• Efficiency : the use of resources in providing something, e.g., cost per listserv 
transaction 

• Effectiveness : how well something is done or the degree to which a service met its 
stated objectives, e.g., number of books cataloged that actually circulated in their 
first year on the shelf 

• Impact : how a service made a difference in some other activity or situation, e.g., 
grades of freshmen who took a library bibliographic instruction program as 
compared to grades of those freshmen who did not take the program. 

Almost all measures found in the literature can be recognized as belonging to one of these types. 
Example measures identified in the literature are grouped in the sections below. 

Extensiveness Measures. Although extensiveness measures are probably the easiest 
measures with which to deal, most of the literature agrees that they are not at all sufficient for the 
evaluation of information systems. These measures are often referred to as traditional measures. 
Berggren, Gustafsson and Lindh (1992) describe the RTK or real-time kernel performance 
evaluation. Bock and Klepper ( 1992) describe the Function Point analyses to mezisure the output 
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of infonnation systems. Evans et al. ( 1988) describe the Krobock Methodology to measure the 
utility of information systems. Rubin (1991c) uses measures such as the number of lines of code, 
the number of lines of executable code, and penetration measures (how many people who could 
use the system are actually using it). 

Efficiency Measures. Kanter ( 1987) and Scudder and Kucic ( 1991) describe the Return On 
Investment (ROI) ratio. Campbell (1992b) calculates how technology investment contributes to a 
bank's net income. Oman and Ayers (1988) and Arkush and Stanton ( 1988) each do a cost benefit 
analysis, the former by looking at the different kinds of productivity (office productivity, 
organization productivity) and the latter by using sample values so things like user satisfaction can 
be taken into account. 

Effectiveness Measures. Effectiveness measures for reference services can be found in 
Bunge (1990) in the form of success rates of reference question answering as perceived by 
librarians and patrons. Murfin and Gugelchuk (1987) assess the reference outcome in terms of a 
patron report. Green ( 1988) and Gorman ( 1987) both measure reference effectiveness using 
different reference criteria for staff, bibliographic instruction, and online services. Fleit ( 1994) 
provides questions to aid self-assessment for campus IT. In answering the questions academic 
computing centers are comparing themselves to an ideal situation. Slevin, Stieman, and Boone 
( 1991) describe using the critical success factor (CSP^ process as a performance measurement. 
Evans et al. (1988) describe the Bailey and Pearson methods to measure user information needs. 
Rubin (1991a) describes failure intensity (number of failures per time period), technical quality 
(design strength in relation to operability and maintenance) and functional quality as effectiveness 
measures. 

Impact Measures. In a case study, Wilson (1993) measures the impact of technology in the 
Peakview Elementary School. By doing a longitudinal study, the changes in students’ and 
teachers’ behaviors and attitudes toward technology could be measured. 

Two general kinds of measures emerge from the literature: system-based measures and 
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user-based measures. System-based measures, such as Real-Time Kernel performance or 
response time, are sometimes referred to as traditional performance measures and tend to measure 
the technical features of a system. Although system-based measures are more straightforward and 
easier to quantify than user-based measures, such as user satisfaction, system-based measures are 
generally believed to be less adequate in evaluating the impact of information systems. 

There are many questions concerning the degree to which business ^proaches are 
transferable to higher education. It has become clear that traditional productivity measures do not 
transfer. Koenig and Wilson (1996) suggest that "the standard measure of productivity in a service 
environment is fundamentally doomed to be increasingly meaningless” and that to have meaningful 
measures we must totally overhaul the ways in which the productivity of services is calculated (p. 
3). Some of the basic concepts of information systems measurement might be transferable but 
specific measures will most likely differ since the higher education environment differs 
considerably from that of business. 

Traditional system performance measures or technical measures (Baccelli and Fayolle, 
1983), such as the number of dialup lines, are described in the literature but the literature also 
suggests that these should either be ignored or used only in combination with other measures (Lane 
and Hall, 1989). 

Overall, the literature on measures of electronic networked resources and services does not 
offer actual measures. Even articles with hopeful titles such as "Yes, There Is a Way to Measure 
MIS Investments" do not go beyond providing general guidelines that one should follow prior to 
establishing the measures. Authors have a tendency to focus on issues related to measures or the 
act of measuring, and shy away from the real issue: the actual measures. Notable exceptions are 
Evans etal., 1988; Fleit, 1994; Rubin, 1991a; Scudder and Kucic, 1991; and Slevin, Stieman, and 
Boone, 1991. 

Heterick (1994, p. 64) expresses concern about the traditional system performance 
measurements which were merely dealing with inputs. "Computer folk have counted lots of things 
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for lots of years." Most of those things had to do with arcane details of how systems "performed." 
Although Heterick calls for measurements of outputs rather than inputs he does not provide details 
as to what kinds of outputs or how these outputs should be measured. 

Part of the literature suggests involving the user in the measurement. McClure ( 1994) calls 
for developing performance measures that have a user perspective so that we can better understand 
"the role, importance, and impact of networked information services in the academic setting." 

Reliable and valid measurement and evaluation in the social sciences is extremely difficult. 
When the measurement and evaluation includes both a behavioral and technological context, it is 
even more difficult. The literature agrees that measurement of networked and computer-based 
services is a complex problem. As shown in the previous review of the literature, the ability of 
researchers to operationalize possible indicators of impacts of networked and computer-based 
services is extremely limited. 

What becomes clear after reading the limited literature on this topic is that little research has 
been done on the topic of measuring impacts on the academic institution. Related literature 
provides us with some insight as to what measures and data collection methods are used in various 
fields. However, the measurement of IT remains a difficult topic. Brynjolfsson (1993) warns us 
that "Even with substantive improvements in our research on IT and productivity, researchers must 
not overlook the fact that our tools are still ‘blunt’" (p. 76). He goes on to say that we as 
researchers should be "prepared to look beyond conventional productivity measurement 
techniques." 

What seems to be needed is a usable combination of measures. Not until one combines 
extensiveness, effectiveness, efficiency and impact measures does one get a complete assessment 
of networking service. And we need a range of measures which assess networks, information 
systems and services from a user perspective. 
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2.3. Assessment of Existing Network Models 

One goal of this investigation was to define the academic networked environment and 
develop a generic and conceptual model of this environment The study team conducted a review 
of existing network models to provide a foundation for this model of the academic networked 
environment Three of the existing models which were found to be relevant to this investigation 
are described below. 

National Research Council Model. The National Research Council (NRC) (1994) 
developed a set of definitions which are useful in the design and analysis of network models. 

They include: 

• Network: A communications system that connects geographically distributed users 
with links and switches as well as control software. 

• Network Functionality: Networks support communications between programs 
running on groups of two or more computers. Such programs implement 
applications that may be running on behalf of one or more individual users. The 
interchange of data between hosts and users takes place as character streams that 
pass between them. 

• Local Area Networks (LANs) and Wide Area Networks (WANs): LANs are 
networks that operate in limited geographic areas, such as single buildings or 
campus settings. WANs connect computers or LANs over larger areas. The most 
unique and extensive WAN is the network of networks known as the Internet. 

The NRC also developed a vision, or model, of what is referred to as the Open Data Network 
(ODN). The ODN is capable of carrying information services of all kinds, from suppliers of all 
types, to a variety of customers, across LANs and WANs, in a seamless, accessible fashion. This 
"openness" in architecture requires relevant standards to be set for the physical infrastructure and 
offered services as well as for oversight and management of the overall network. 

The NRC model (Figure 2-1) offers a highly technical structure of layered connectivity 
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FIGURE 2-1. NRC MODEL 
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from the Bearer Service (Layer 1) through the Transport Services and Middleware Services 
(Layers 2 and 3) into the Applications Area (Layer 4). The path of progression within this model 
is through all levels in succession until the application is addressed. The model has not been 
operationalized with the intent to provide quantitative measurements within each layer. Although 
users are mentioned often within the NRC model, in terms of authenticating, verifying 
identification, security, and control, users do not represent a fundamental component, nor a unique 
layer, within the model. 

The NRC model provides useful ideas on the composition of highly technical models, 
especially concerning the interaction of the transport and applications layers. It does not, however, 
offer insight into assessment or measurement of the effectiveness of networks of this type. 

Cross-Industry Working Team Model. The Cross-Industry Working Team (XIWT) of the 
Corporation for National Research Initiatives (CNRI), has developed An Architectural Framework 
for the National Information Infrastructure. This framework is the collaborative work of 
individuals from 41 of America's telecommunications corporations and societies. 

This document provides a vocabulary and context for discussing an architecture for the 
Nil, identifies some necessary fundamental Nil components and services, and examines ways to 
expand and evolve the infrastructure. The aim is to allow communications, computer, and 
information providers to work separately while implementing elements of a common vision. The 
principles driving this model, called the Functional Service Model, include a feature-rich, open, 
and distributed architecture, based on functionality, trust, and control (XIWT, 1994, p. 12-17). 

The XIWT model includes three component layers; Physical Infrastructure, Enabling 
Services, and Applications. (See Figure 2-2.) These component layers are tightly bound with three 
aspect layers: Function, Trust, and Control. As in the NRC model, functionality implies an open 
architecture, while "trust" includes the concepts of security, integrity, and performance assurance. 
The control layer includes concepts of management, service, and flexibility, along with the ability 
to measure, and to provide accounting functions. Although this model emphasizes performance 
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measurements, in the form of statistics and descriptions of the component states, it, like the NRC 
model, does not focus on the user as one of its foimdations. 

CSPP Critical Interfaces. The Computer Systems Policy Project (CSPP) is an affiliation of 
chief executive officers of American computing companies that develop, build, market, service, 
and support information processing software and solutions. Since 1989, this group has been 
committed to the development of policy positions on trade and technology issues affecting their 
industry and ultimately the United States (CSPP, 1994). 

The CSPP's Perspective on the National Information Infrastructure: Ensuring 
Interoperability, published in February, 1994, is also of importance in studying the Academic 
Networked Environment. In this report, CSPP proposes an industry-led strategy to ensure 
interoperability. CSPP does not suggest a specific architecture, as the other models do, but 
instead, identifies critical points within the infrastructure that must be interoperable for successful 
functionality. These interfaces, identified by CSPP, fall into four categories: 

1 . Appliance (User) to Network: the interface between an information 
appliance, operated by a user, and a network service provider. 

2 . Appliance (User) to Application: the applications programming interfaces 
(APIs) between users and applications. 

3 . Application to Application: the protocols that an application, service, or 
system uses to communicate with another application, service, or system. 

4. Network to Network: the interfaces amaig and between network service 
providers. 

CSPP believes that ensuring openness in these interfaces will advance the following critical 
Nil goals: 

• To allow users and providers to transmit information smoothly throughout the 
network. 

• To enable wide access by users and providers of information zmd services. 
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• To stimulate competitive markets for Nil products and services. 

The uniqueness of the CSPP report is the detailed attention to the user community. The focus is on 
how the Nil will revolutionize the way individuals will relate to one another by enabling them to 
work together to access, generate, store, process, transmit, and receive text, data, and images 
anywhere at any time. 

The CSPP model (Figure 2-3), proposed for the national infrastructure, could be adapted to 
a smaller scale for an academic environment. The same issues of inter-operability between 
network components would still apply. Another contribution of this model is the focus on the user 
and the critical interfaces between users and other network components. 

2.4. Review of Existing Data Sources 

Although there are few existing models and measures which can be applied to the academic 
networked environment, there are a number of sources which offer relevant data on academic 
institutions and IT. Such data may be useful in describing the academic networked environment 
and in developing measures for academic networks. The major sources of relevant existing data 
are described below. 

General Demographic Data on Universities. The National Center for Education Statistics 
annually collects statistics on all universities through IPEDS (Integrated Postsecondary Education 
Data System). There are no variables specifically related to IT in IPEDS. However, several types 
of demographic data might enter into the computation of performance measures: 

• Enrollment in the institution 

• Total/full-time students/part-time students 

• Students by level (undergraduate/graduate) 

• Students by field of study 

• Degrees conferred: by level (bachelor's, master's, doctor's); and by field of study 

• Compiosition of staff: total and FTE; and by categories (profession^ (executive. 
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administrative and managerial/Faculty/TA and RA/non faculty professionals) ; and non- 
professionals (technical and paraprofessional/clerical and secretarial/skilled crafts/service and 
maintenance)) 

• Expenditures per FTE student 

• Data on IT in Universities 

IT Surveys. A number of surveys concerning the use of IT in academic institutions have 
been conducted recently by groups such as: 

• The Center for Scholarly Technology at the University of Southern California (Fall 

1993) 

• The American Association of State Colleges and Universities (AASCU) (Spring 

1994) 

• The Corporation for Public Broadcasting (CPB) (Spring 1994) 

Each of these survey instruments is aimed at a slightly different population, all broader than the 
sample for this study, and has a slightly different purpose and approach. However, there is some 
overlap between these populations and the sample for this study. Variables specifically pertaining 
to the networked environment are : 

• Issues considered as important in strategic plaiming 

• Type and percentage of on- and off-campus facilities linked to the network 

• Services offered on the network (e-mail, library catalog, Campus-Wide Information 
System (CWIS), file server, etc.) 

• Internet access for faculty/staff/students 

• Fee policy 

• Availability of support and training 

• % of student/staff/faculty having access to and using e-mail 
Other related variables are 

• Number of machines available, by categories of users, type of facility 
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(cluster/classrcxjm/office), and ownership 

• Number of distance-learning courses offered 

Most variables measure only availability of services. E-mail seems to be the only kind of 
service for which there is an attempt to quantify actual use. Very few questions deal with the 
impacts of IT on teaching and research. Those which do only call for the subjective assessment of 
the respondent measured on a Likert scale, and are usually phrased in very general terms (for 
example, "How well has your institution used technology to improve the overall quality of 
education at your institution?" ). 

use Faculty Survey on The Role of Information Technology in Instruction and 
Scholarship. This 1994 survey of faculty in two- and four-year colleges, evaluates use and 
benefits of IT in teaching and research. Network-related extensiveness indicators are: 

• Existence of LAN access, network connections, and modems for computers at 
home and in the office 

• Self-assessment of IT skills, including using online information systems, using e- 
mail, and using the Internet 

Use indicators include: 

• Use of Internet services (e-mail to other campuses/databases or library catalog 
searches/access to government databases/participation in a listserv): 

never/ monthly/weekly/ daily 

• Network activities since beginning of current term (e-mail to foreign country 
colleague/e-mail to student/FTP/access materials from another campus/e-mail 
discussion/felt like receiving too much e-mail): no/once/more than once 

• Instructional materials and techniques used for courses taught this term: e-mail to 
students, materials found via Internet, audio- teleconferencing 

Impact indicators include: 

• Assessment (on Likert scale) of importance of IT resources: includes access to the 



O 

ERIC 



37 



34 



library catalog, access to online information resources, e-mail to colleagues (on 
campus and to other campuses), access to materials via Internet 

• Assessment (on Likert scale) of benefits of IT in the past five years and in next five 
years in the following areas: 

• Productivity as a teacher 

• Productivity as a researcher 

• Reputation of institution 

• Quality of programs 

• Contact with colleagues 

• Opportunity to work with other departments and campuses 

• Access to information resources 

This survey also includes questions related to faculty outputs such as 

• Have you, during the past 3 years, received a recognition award for teaching? 

• What professional activities have you been engaged in during the past 3 years (# of 
courses taught, presentations at conference, articles, chapters in books, 
monographs)? 

These output measures may then be correlated with measures of network use. 

NACUBQ Benchmarking Project. The National Association of Colleges and Universities 
Business Officers (NACUBO) collects annual benchmarking data on the administrative functions 
of higher education institutions. Among the 48 research institutions participating in IT 1993, 18 
(3 private, 15 public) are in the population for which this study was intended. For the moment, 
most indicators measure costs and services levels, because these data are more readily available 
than others. The project covers 36 IT indicators, including five which are specifically related to 
networking: 

• Number of institutional networks 

• Networking cost per network connection 
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• Number of users networked when the current network architecture plan is 
implemented 

• Cost of five- year network architecture plan 

• % of faculty, staff and students who are e-mail users 
Other indicators deal with IT in general. The most significant ones are: 

• Cost per user (total/central IT/decentralized IT ; operating/personnel/capital costs) 

• Number of training hours per user 

• Number of departmental PCs and workstations per staff FTE 

• % of students who own their own PCs 

CAUSE Institutional Database. CAUSE aimually collects statistics on computing services 
from all its members and maintains a database of this information. CAUSE Data collected in the 
following areas may be useful in describing the academic networked environment: 

• IT management (level of integration of the various computing, information and 
telecommunication functions) 

• IT staffing by function (academic computing/administrative 
computing/telecom/networking) and type of work (management/ 
development/operations/user services) 

• IT budget by function 

• Strategic plaiming: functions included in IT plans, areas where policies or plans are 
in place. 

• Types of "emerging technologies" implemented or planned 

Network related variables are very similar to the ones used in other surveys. But several of them 
are more specific: 

• % of each user group (students, faculty, stafO having access to and using regularly 
e-mail, gopher, world wide web 

• % of institutionally owned academic micros/ workstations networked 
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• % of institutionally owned administrative micros/workstations networked 

• Ratio of institutionally owned academic micros/workstations to people in each user 
group 

The section on academic computing uses several indicators to estimate the use of IT in teaching: 

• % of faculty making use of software in classroom 

• % of faculty making use of multimedia software in classroom 

• % of faculty incorporating use of computing in the curriculum 

• % of classrooms equipped for computing or multimedia 

Summary. All of the data sources included here have some elements which apply to 
measurement in the academic networked environment The CAUSE database seems to be the best 
source of information concerning the following aspects of the networked environment: 
organization, resources available, and use of theses resources. The USC Faculty Survey is the 
best source of information concerning the level of use of network resources. And IPEDS provides 
useful demographic statistics on universities. Other unique contributions from each source include: 
NACUBO’s data on costs, USC’s data on participation of each academic field, AASCU’s distance 
learning data, and CPB ’s impacts of the academic network on the broader community (participation 
in a community network and access to the institution's resources by outside users). However, 
none of these instruments measures adequately and comprehensively: 

• How well the network performs in term of access (problems of slowness, refused 
connections etc.), ease of use, success of use 

• To what extent specific kinds of network tools and resources are used 

• For what specific purposes network tools and resources are used (teaching, 
research, homework, communication, personal information need, leisure) 

• What the impacts on research and teaching performance are. 
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2.5. Analysis of IT Plans 

In addition to reviewing the literature, existing network models, and existing sources of 
relevant data, the study team examined a group of university computing strategic plans to identify 
existing approaches and measures associated with the academic networked environment. This 
analysis supplements the other resources reviewed in that it specifically represents the perspective 
of academic computing professionals. The research questions which guided this analysis of the 
strategic plans are as follows: 

• What are the components of the academic networked environment? 

• What are the impacts, both actual and anticipated, of networking in the academic 
environment? 

• What, if any, evaluation of the academic networked environment is being conducted 
or planned? 

Twenty-one plans were selected, as described below, and a content analysis was performed 
to identify models and measures, as well as key issues, themes, and concepts. Four criteria were 
used for selection of the plans: 

• The document is for an institution which meets the following criteria: Carnegie 
Research University I or II, library is a member of ARL 

• The document is an actual plan (not a generic description of the planning process) 

• The document is recent (published in the 1990s) 

• If possible, the document should deal with computing services as a whole (i.e., 
including academic computing, administrative computing, and network- 
telecommunications). However, partial plans are acceptable as long as they are not 
concerned with technical objectives only. 

Copies of the plans were obtained from the CAUSE Information Resources Library, and 
also from participants in focus groups which were held as part of this study. For three institutions, 
two plans were included, either because they had complementary purposes or because several 
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years hjid elapsed between both documents. 

The plans varied in length from 7 to 219 pages and they were extremely heterogeneous in 
content and emphasis. For instance, some plans analyzed the current strengths and weaknesses of 
the institutions, other did not. Expected impacts were spelled out in some plans but only implied 
or taken for granted in others. Also, some plans were not comprehensive. Two plans dealt only 
with administrative computing, two others dealt only with academic computing, and another dealt 
solely with the campus network. 

The initial coding scheme was developed from a content analysis of two plans, then tested 
with two other plans. The final scheme includes the following major categories: 

• Benefits 

• Objectives 

• Stakeholders 

• Resources 

• Organizational structure 

• User involvement 

• Evaluation 

This scheme was adjusted during analysis of the 17 remaining plans. Category definitions were 
refined and a number of subcategories were identified and defined. The complete coding scheme is 
presented in Appendix A. Definitions and examples for each category and subcategory are 
presented in Appendix B. 

The content analysis of the plans is discussed below, for each of the major categories, 
identified. This analysis is best seen as a preliminary assessment of the major areas discussed in 
the various plans. 

Benefits. Potential benefits of the networked environment exist in all areas of activity of a 
university, but to differing degrees. There is a consensus that IT increases research productivity, 
mainly through increased access to data and increased cooperation. In both cases, this is true 
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within the institution (e.g., remote access to the library catalog, collaboration between departments) 
and outside of it (e.g., access to a supercomputing center, searches on remote databases, 
collaboration with researchers at other institutions). Although it is sometimes difficult to 
distinguish between them, it seems that benefits to the research process are more emphasized than 
benefits to the communication of research results (i.e., electronic publishing) 

In the administrative area, there is also a consensus on the two main impacts: improved 
access to institutional data - both by staff and by "end-users" - and increased efficiency of 
administrative processes. The other aspects noted by some institutions, neimely cost reduction, 
simplification of the organizational structure, and better decision-making, can be seen as 
consequences of these two primary benefits. 

In the area of teaching and learning, there is a greater variety of expected benefits. Richer 
learning materials, in the form of multi-media and simulations, are mentioned most often. 
Interestingly, access to databases, which emerges as a major benefit for researchers, is very rarely 
mentioned as a benefit for students. Two additional benefits, directly linked to the network, are 
increased communication between students and faculty, and development of distance learning. 
Other benefits noted by only a few institutions, namely exposure to diversity and preparation for 
future jobs, are those not directly related to the teaching process, but more to general outcomes of 
education. 

Finally, benefits related to the service mission of the university are mentioned rarely and in 
a vague way. There seems to be a belief that the networked academic institution can offer better 
services to its surrounding commimity, but it is not very clear how. 

Objectives. On the whole, the plans are very similar in terms of objectives. Almost all of 
them include: 

• The development of policies, guidelines, and standards, made all the more 

necessary by the trend toward a distributed environment. In most cases, this seems 
to be considered only at an internal level. However, a few institutions also discuss 
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standards and policies at the national level. 

The development of security procedures 

An adequate infrastructure, including interoperability between systems, coimection 
of offices, dorms, classrooms and sometimes homes to the campus network, with a 
particular emphasis on the equipment of classrooms, and coimection to national 
and international networks 
Adequate user support and user training 

The development or implantation of information servers of different kinds: 
administrative systems, CWIS, library catalog, bibliographic databases. 



There is also agreement among the plans on objectives to encourage the use of IT in 
teaching. These include: the establishment of faculty rewards such as sabbaticals, grants, 
copyright fees, and integration of IT aspects into the tenure process; and the formal integration of 
IT into the curriculum, both by the development of courseware and teaching of IT-related courses 
as part of a general education core. Fewer institutions identify the encouragement of student 
purchases of computers as an objective and those that do mention it differ on how this should be 
addressed. 

Some institutions state that every student should have his or her own equipment, and 
support to public computer clusters is temporary. Other institutions maintain a commitment to 
public clusters which guarantee universal access whatever the economic means of the students. 

Finally, only a handful of universities (all of them state institutions) include a specific 
objective related to services to the larger community, either through distance learning or through 
support of a community network. Only two institutions list as an objective the provision of 
adequate IT tools for disabled students and faculty. 

Stakeholders. Students, faculty, staff, and computer services were identified as interned 
stakeholders in all the plans. External stakeholders identified are primarily of two kinds: private 
companies and government agencies. Both can be considered either as sources of funding or as 
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partners in research projects, especially in the area of supercomputing. Often, the two roles are 
combined. The institutions which include service to the community in their objectives also 
consider K-12 schools with which they collaborate cis stakeholders. 

Resources. Not surprisingly, most institutions have very similar concerns in the area of 
resources. The major concern is a sufficient and stable fimding bcise, which seems in jeopardy at 
certain institutions. Some plans discuss establishment of a specific student IT fee or new rules for 
charging departments for services. Other resource concerns include: an adequate, well-trained, and 
motivated staff, sometimes threatened by competition from the private sector; and continuous 
purchase and maintenance of adequate hardware and software. Issues related to hardware and 
software are access to supercomputing resources, either locally or through a regional consortium, 
and migration from a mainframe to a client/server environment 

Not all plans are concerned with budget requirements and procedures and resource 
allocations. Those which are focus on the sharing of responsibilities between central computing 
services and individual departments, a problem which is much more acute in a distributed 
environment than in a centralized one. 

Organizational Structure. The most important issue related to organizational structure is the 
balance between centralization and decentralization. Migration to a client/server environment leads 
most institutions to emphasize decentralization. However, many of them also stress the need to 
maintain a central core, to: 

• Support common infrastructure services (network for instance) 

• Limit costs (purchase of general site licenses) 

• Guarantee interoperability with guidelines and standards 

• Coordinate the planning process for IT, and insure its integration into the 
University-wide planning process 

• Provide specialized user support (while day-to-day support is generally seen as a 
responsibility of individual units). 




45 



42 



Interestingly, a good proportion of the plans recognize the need for good coordination with other 
support services, the library being mentioned most often, especially for the development of 
information servers. 

User Involvement. Most institutions clearly see themselves as "demand-driven" and seem 
committed to incorporating user needs into their decision processes. Indeed, a few plans devote a 
full section to a detailed analysis of user groups and their specific needs. However, except for the 
traditional advisory committees, which often include only faculty, there is no consensus on the best 
approach to obtain user input. A wide variety of methods are mentioned, including: open campus 
meetings, evaluation of computing services director by faculty, input from people not familiar with 
IT, and online user input. Some of these methods, such as surveys and meetings, can provide 
only periodic input while others, such as online user input and liaisons, can provide data 
continuously. The establishment of user liaisons in each department is used most often. 

Evaluation. Evaluation is clearly not a major concern in most strategic plans. Often, it is 
not considered at all, or only in very general terms. A few institutions, however, include a section 
analyzing their current strengths and weaknesses, showing that they do indeed assess their 
performance. Several plans, however, include a section recommending the implementation of an 
evaluation process, either in general or focusing on specific aspects, i.e., cost analysis, assessment 
of new alternatives, effectiveness of IT currently implemented. A few plans include details on 
possible methods of evaluation, such as: 

• Review of policies 

• Peer comparison 

• Satisfaction surveys 

• External reviews 

• Assessment by users of the extent to which goals of strategic plans are met 

Measures used or suggested are all measures of extensiveness, dealing either with access to 

network resources or with use of these resources. They include: 
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• Bandwidth of network backbone 

• Connectivity: extent to which all campus locations are connected to network 

• User access: extent to which all members of the university community have actual 
access to the network 

• Number of listserv messages per month 

• Ratio of computing seats per student 

• Number of online transactions in administrative system. 

In most instances there is little explanation of how such measures were used or if they were 
actually implemented. 

A few reciuxent themes emerge from this analysis. For example, impacts and objectives 
are: 

• Clearly defined and common to all institutions in two areas: research and 
administrative support 

• Well defined but with less consensus in the teaching/leaming domain 

• Still very vague as far as service to the broader community is concerned, except in a 
few state universities committed to distance learning and to the support of 
community networks. 

In all areas, the main impacts are always linked in some way with "increased communication" or 
"increased access to data and computing resources. " Both communications and access to data and 
resources occur within the institution (e.g., cross-departmental collaboration, access to institutional 
data such as grades, schedules, and financial aid) and with other institutions, academic or not 
(e.g., collaboration with researchers in other locations, access to remote databases, and electronic 
submission of grants). The possibility that the academic networked environment might have 
negative as well as positive impacts is never considered. 

Institutions struggle with the necessity of finding a balance between centrali2ation and 
decentrali2ation as the technology switches to a distributed mode. This affects : 
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• The development of policies and standards 

• Budgeting 

• Staffing 

• Hardware and software purchase and maintenance 

• User training and support 

Most of the plans agree that user involvement is necessary in principle. However, except at 
a few institutions, there are not many formal channels of user input into the development of 
campus technology, especially as far as students are concerned. Evaluation of technology, 
applications, or networking is rarely a major concern. There is no consensus on the method of 
conducting it, and measures used, if any, deal only with extensiveness. 

2.6. Summary 

Throu gh the various efforts described in this background section, the study team developed 
a foundation for this investigation. In the process of developing this foundation, the following 
became evident 

• There are no existing measures of network performance or impacts. 

• There is no generally agreed upon model of the academic networked environment 

• There are several sources of data on academic institutions, although none of them 
provides information on network resources, activities, or funding. 

• Although many institutions have developed formal plans for network development, 
the content of those plans varies widely. 
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Notes to Section 2 

1. “An audience is an individual or group that will be using measurement for decision making. A 

stakeholder is an audience that must buy in to make the measurement program work.” (Rubin, 
1991c, p. 79) 

2. Here one establishes what the different parts of an organization need to do in order to function 

well. 

3. Net interest income, noninterest income, operating expenses, and credit quality. 

4. Part of the operating system that manages memory, disk operations, and the file system. 

5. The ratio of the output of a process to the input used. 
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3. METHODOLOGY 



3.1. Introduction 

Because the focus of this study, assessment of academic networking, is an area in which 
little previous research has been conducted, the study team adopted an exploratory approach here. 
That is, the object was to discover and describe the current nature of network assessment and 
related attitudes, activities, behaviors, and constraints, without reference to a pre-existing research 
framework. Likewise, the goal was to characterize rather than quantify the variables of interest in 
this situation. For these reasons, a variety of qualitative methods were used for data collection. 
They included: focus groups, interviews with individuals, small group discussions, and site visits. 
In addition, several survey instruments were developed, an expert advisory panel was established 
to review interim results, and an ongoing electronic discussion group was set up to distribute 
project updates and to solicit feedback. Interview transcripts and notes from observations and 
discussions were analyzed by content analysis. An overview of the methodology which was 
followed in this study is presented in this section, and details concerning the application of each 
individual data collection and analysis method are presented in the Results and Discussion section 
(4.). 

3.2. Data Collection and Analysis 

The collection and analysis of data for this study was done in an iterative fashion. That is, 
the data collected in one stage were analyzed and used to inform the data collection in the next 
stage. Following is a brief summary of data collection activities. 

• A focus group of academic computing administrators and staff from a selected 
university was conducted to explore the study topic. 

• A second focus group and a small group interview were conducted at a national 
meeting of individuals involved in academic computing. 
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Results from the first two focus groups were presented at a conference of academic 



computing professionals and feedback was solicited via a small group discussion 
following the presentation. 

The study team then made site visits to two academic institutions in order to explore 
in depth the issues which emerged from the ^lier focus groups, interviews, and 
discussions. 

Once the data from the site visits were analyzed, preliminary conclusions from the 
research up to that point were tested, and issues were further investigated, in a 
focus group of individuals from the same university where the initial focus group 
was conducted. 

Final results were presented at another national conference of individuals involved 
in academic computing and feedback from the audience was solicited. 

A final focus group was conducted at the conference where the results were 
presented. 



Additionally, the study team conducted an ongoing electronic discussion among interested 
individuals concerning the interim results of the investigation. Feedback from the discussion 
group was incorporated into the evolving methodology. Also, periodic reports from the 
investigation were submitted to an expert advisory panel which also provided feedback which was 
incorporated into the evolving methodology. 

3.3. Population and Sample 

The population of interest in this study includes colleges and universities which are actively 
involved in the development and provision of networked resources and services on their campuses 
and the connection of their institutions to entities outside of their campuses via the Internet. Initial 
criteria for sample selection were: the institutions selected had to form a homogeneous group in 
terms of main institutional characteristics and mission; and each had to have an academic network 
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already highly developed and in use. These two criteria were met by focusing the study on 
research universities, as defined by the Carnegie Foundation for the Advancement of T eaching. 

The foundation classifies all higher education institutions in the U.S. according to two main 
criteria: the range and number of degrees offered and the comprehensiveness of the institution's 
mission. In the Carnegie classification. Research Universities are grouped in two categories : 

• Research Universities I are those which: offer a full range of baccalaureate 
programs, are committed to graduate education through the doctorate, place a high 
priority on research, grant 50 or more doctoral degrees each year, and receive $40M 
or more in federal support. In 1994, there were 88 institutions in this category. 

• Research Universities II are those which have the same characteristics as above, 
with one exception: they receive less federal support (from $15.5 to $40M). In 
1994 there were 37 institutions in this category. 

The sample was further narrowed by limiting it to Research Universities I and II which 
were members of ARL. ARL has 1 19 members, of which 109 are university libraries (including 
13 from Canada). New members are admitted by invitation, and membership is open to major 
university libraries whose collections and services are broadly based and to certain other libraries 
whose collections are recognized as having national significance. Since ARL tends to be involved 
in issues related to networking, especially through its participation in CNI, membership in this 
association was determined to be an indicator that an institution has interests and experience 
relevant to this study. 

Of the 125 Carnegie Research Universities I and II, 87 are members of ARL. Seventy-one 
are classified as Research I, and 16 as Research II. It is these 87 institutions for which the results 
of this investigation will be relevant. Individuals who participated in this study, as focus group 
members, interviewees, and advisors, were affiliated with institutions which are among this group 
of 87, although not every institution is this group is represented. 
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4. RESULTS AND DISCUSSION 

4.1. Introduction 

Results from the various data collection and analysis activities are presented here in 
chronological order, thus demonstrating the evolution of the project as one stage of data collection 
and analysis informed the next. The project began with two focus groups and a small group 
interview to collect data on the current state of academic networking and assessment. Following 
these initial data collection activities, a small group discussion was conducted to review preliminary 
project findings. The study team then made site visits to two institutions where we conducted 
interviews with individuals and small groups. Another focus group was held, feedback was 
collected after a conference presentation, and a final focus group was conducted. 

4.2. Initial Focus Groups 

As shown in the preceding review of the literature, there are few examples of models of the 
academic networked environment and few reports of attempts to identify measures of the impacts 
of networking on academic institutions. This being the case, the first step was to do an empirical 
assessment of the current state of academic networking. This was done via focus groups at two 
different locations. The information gathered in these focus groups would then be used to guide 
the collection of data during the site visits. Anticipated outcomes of the focus groups included: 

• Identif y major issues in academic networking and existing evaluation 
methodologies and measures of impacts of academic networking. 

• Obtain insight into the decision processes associated with the implementation of 
network technologies and services. 

• Identify the groups involved in, or having an influence on, those processes. 

• Identify existing models of the academic networked environment 

• Validate the findings from the literature review which indicated a very sparse 
amount of information on the topic. 




53 



50 



• Generate views and perspectives that would be useful to the study team in terms of 

( 1) developing a model of the academic networked environment, and (2) developing 
measures to assess this networking environment. 

• Inform the future development of the project's methodology. 

These focus groups generated extremely useful information that helped define project activities and 
suggest strategies for accomplishing broader project goals. 

The first focus group was held at a major private university in the northeast on November 
21, 1994, and included members of the school’s computing services group. The study team 
designed the focus group primarily as a pretest for a later focus group, but the results from the 
pretest were very valuable and have been incorporated as part of the project findings. 

The study team conducted the second focus group, as well as a small group interview, at 
the 1994 Fall Meeting of CNI in Orlando, Florida. (The study team originally plaimed to conduct 
two focus group sessions at CNI, but due to scheduling conflicts with other conference activities, 
one session was attended by only two participants. Thus, the study team conducted the session as 
a small-group interview.) This conference was especially appropriate for the data collection 
activities as members comprise three key groups of interest to the study: CAUSE (academic 
computing professionals), EDUCOM (higher education technology professionals), and ARL 
(academic research library professionals). 

In both locations the study team followed standard methodological procedures for 
identifying participants and conducting the focus group sessions (Krueger, 1988). Selection of the 
university participants was based on criteria that included; knowledge of the university network, 
experience in either developing or managing that network, and an interest in participating in the 
focus group. Criteria used for the selection of the CNI participants included: upper-level university 
managers responsible for administering the network at their institutions; familiarity with national 
networking issues; interest in the measurement and evaluation of networked services; and ability to 
attend the CNI conference. 

O 
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For all sessions, the study team prepared a set of questions to guide the discussions. The 
topics covered in the university session (see Appendix C) informed the topics that the study team 
selected for use at the Orlando sessions (see Appendix D). At all sessions, a study team member 
distributed questionnaires that requested demographic and other background information (see 
Appendix E). 

Transcripts and notes from the focus groups were content analyzed to identify key issues. 
The research team members who acted as moderators analyzed the transcripts independently and 
summarized the notes taken. The summaries were compared and are integrated in this report. The 
following sections provide a summary of the issues discussed at the university focus group session 
and the CNI sessions, and conclude with an analysis of the responses to the questioimaires 
completed by participants in all the sessions. 

4.2.1. University Computing Focus Group 

Seven employees from the university’s academic computing services (ACS) group 
participated in the focus group, which lasted approximately two hours. The key issues which 
emerged during this session were: 

The Network as a Utility. The participants saw the network as a utility such as central 
heating or the telephone. One participant used the analogy, “We are like plumbers: water or bits, 
must go through the pipes or wires. Plumbing is the same as wiring, but our systems are more 
complex." This view, that the system is the same as heat and electricity, makes it unnecessary to 
justify the existence of the network. 

"We believe in the technology, and believe that the outcome is positive. This is accepted 
technology. You don't ask questions. Just use it, like telephone service." This analogy removes 
responsibility by these professionals for ensuring that the network is used efficiently. Just as there 
is little concern for how the telephone is used, unless illegally, the participants are not overly 
concerned with the lack of training among users. 
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Evaluation. To the participants' knowledge, there are no formal processes to evaluate the 
network. Their definition of a good system is one with seamless integration, "the less you notice 
the network, the better it's running. If you cutoff the network for one day, the users will see its 
impact." An invisible system does not generate much feedback. The participants are not aware of 
the user's needs. "We don't know what the faculty does, or how they use the system, unless it 
crashes and bums." 

When pushed to answer how to judge the efficacy of the network, they discussed system 
standards, such as the amount of time the system is running, or meeting the manufacturer's 
standards for the ethemet. "We can predict how often the system is up, it's somewhere in the 
99.9% range, but that doesn't mean that people are using the system successfully. The user 
doesn't care that the reason they can't use the system is user error." 

Daily reports are compiled listing the busiest networks, saturation times, and user 
complaints. However, these reports are not categorized or kept To these participants, reliability is 
the main issue to evaluating a network. 

The participants acknowledge informal peer evaluations. They are very aware that the 
university must stay technologically competitive. The marketing research recmitment data showed 
that students do not base their decision to attend this university on its information infrastructure. 
However, if the university was not technologically competitive, recruitment would be affected. 
Benchmarking against peer institutions or "keeping up with the Joneses" appears to be the most 
effective method used by these participants to evaluate the network. These informal comparisons 
are performed by different stakeholders within computing services: "The higher up in the 
hierarchy, the more the comparison is of the overall system. The lower in the hierarchy, the more 
one is interested in the nuts and bolts." 

User Involvement. When the participants discussed the user, they usually meant faculty 
members. When asked who is responsible for assessing the impact of the network at the 
university, the participants answered that the faculty and the deans are. Although the faculty is 
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responsible for assessing the impact, the participants freely acknowledge that the faculty is poorly 
trained to do that Another problem that they perceive is the unrealistic expectation of the faculty 
that they can use the system efficiently with no time investment. 

As the plumbers, their responsibility is to build and maintain the system. They don't know 
how the system is being used, unless the system fails. When the system fails, they receive user 
input in the form of complaints. The participants state that only the faculty and the deans know if 
the system is being used effectively. 

The information infrastructure is partially demand-driven. There are some services which 
must be updated or changed because of user overload. There are also some services which the 
user is comfortable using and is resistant to changing. 

It is very difficult for faculty with only a basic knowledge to assess the impact of the 
network. The faculty have different accesses to the network, some have ethemet connections, 
while others have dial-up connections in a central location. Some departments are heavy users of 
the technology, while others are not Faculty are not uniform in their use or knowledge of the 
technology which is available. Simply stated, the faculty is not in a position to assess the impact of 
the network on the academic environment 

Vendors. The information infrastructure is partially vision-driven. It would appear that 
vendors should attempt to supply the vision by promising applications, impacts and outcomes 
linked to their product. However, very few vendors are interested in the university market 
focusing instead on corporations. The technology is so state-of-the art, that the vendors are not 
sophisticated enough to promise impacts. They have difficulty thinking of applications. From the 
perspective of the participants, vendors believe that their product should be purchased because the 
technology is good. 

In addition to these key issues, the focus group participants identified the following barriers 
which limit them in building a better networked environment: 

User Training and Computer Literacy . User training and computer literacy barriers focused 
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on faculty. There is no accurate way to measure computer literacy. Some participants suggested 
self-evaluation surveys and tests to determine computer literacy. It is important to train the faculty, 
as they will set the tone on campus and the systems will be used efficiently and effectively. 

Budget. In these times of decreased em-ollment, decreased federal and state funding, and 
down sizing, budgets are being slashed. Investing and maintaining information technology is very 
expensive. There are times when the users expect ACS to be venture capitalists. The participants 
are uncomfortable in this role. If the departments are unwilling to make the technology investment, 
then ACS does not want to invest either. As budget increases become more competitive, the need 
to justify expenditures will become critical. 

FiinHamp.ntal Changes Without Disruption and Interoperability . One of the fundamental 
goals of ACS is seamless integration, therefore, it is very difficult to make any changes to the 
system without disrupting the network. These disruptions must be made at times of limited user 
access in order to maintain the appearance of transparency. When planning changes or 
expenditures, it is important that all the systems function together. 

Human Resources . One participant stated that "human resources are a bigger obstacle than 
budget constraints." It is difficult to attract and keep a staff with the expertise and devotion to 
maintain the system. It is also necessary to have a staff who can be both demand- and vision- 
driven. 

Space and Distributed Nature of ACS. The participants felt that space was a problem 
because ACS is in nine different locations on campus. They would like a centralized space, in 
order to be involved in the every day decisions of ACS. 

Impact Assessment. When the moderator asked what types of statistics could be gathered 
to assess the impact of the network, the participants quickly listed the following: 

• e-mail traffic 

• the number of local listservs and participation 

• cluster utilization 
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• the number of courses with a computer component 

• dial-in statistics 

• the number of network connections 

• the number of times that CWIS is accessed 

• and the timeliness of CWIS. 

A key theme in this session was the difficulty in separating the Intemet/NREN from the 
Campus- Wide Information System (CWIS), or other networking services. The seamless 
integration of the network has made the system transparent, and the user may not know whether he 
has accessed the campus system or the Internet. The parameters of this study were broadened at 
this point, because the participants demonstrated that the system is ubiquitous. 

Another key finding was that there are no impact measures gathered by these computing 
professionals, and that these participants did not think that impact measures were needed. 

However, in discussions about budget limitations, they acknowledged that justifying the impacts 
would help justify expenditures. 

4.2.2. CNI Focus Groups 

Twelve academic computing professionals participated in the CNI focus group and small- 
group interview. The key issues which emerged during these CNI data collection efforts are 
presented here in three major categories: 

• Drivers of the development of network technologies and services 

• Barriers to the development of network technologies and services 

• Measures of the effects and impacts of network technologies and services. 

To a certain extent, these categories are products of the questions asked. For example, participants 
identified measures of the impacts of networking because they were asked how they evaluated their 
networking functions and whether their networks were adequate for the needs of their institutions. 
However, many of the subcategories described below, such as measures of input and measures of 
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output, emerged directly from participants' comments. Also, the moderators encouraged 
participants to share their definitions of terms used in the questions. For example, the moderators 
did not give participants a definition of adequacy but rather asked participants to explain how they 
measured the adequacy of their networks. Participants' definitions emerged from those 
explanations. 

Drivers of Network Development . All of the CNI participants were from institutions where 
the demand for networking capabilities is growing. They identified a number of forces, discussed 
below, which appear to be fueling this growing demand. 

Participants identified two major classes of users — faculty and students — and discussed 
characteristics, expectations and needs of each class. Participants saw user demand for services 
and dissatisfaction with existing services as primary drivers of networking development. 
Participants perceived that a certain level of networking capability, which they couldn't define 
further, is essential to support faculty activities, especially those activities involving collaboration. 
For research universities, a driving factor in getting the network is that the scientific and 
engineering communities can't do business without the network.... A physics department, 
for example, would rebel today if they didn't have a high bandwidth connection to the 
global network. I mean they would be out of business. They communicate constantly with 
people at.. .Fermi lab and CERN... their peers around the world.. ..For research 
universities, pretty much it's not an option. 

In addition to collaborating on research projects, faculty are increasingly collaborating on 
applications for research funding, and need networking capabilities to do so. 

Researchers at the university are not capable of competing for grants without this kind of 
access.... [we] are sort of routinely team responding with one or more institutions. The 
cost of research devices being such that oftentimes universities will pool their resources 
when they respond to an NSF grant request, or something like that. Clearly they wouldn't 
be able to do it without that [without networking], so they would just start losing grants at a 
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high rate of speed. 

For certain kinds of institutions, specifically small and geographically remote institutions, 
networking may be essential to their survival. "The faculty argues that, since they are so resource 
poor at their institution, the only way in which they can.. .get access to resources that will upgrade 
the quality of the little institution is a good networking infrastructure." Thus, it is not at just large, 
research institutions where networking is needed. 

At times, it is not user demand that network providers listen to but rather, what one 
participant described as, "A low-level, or not very high priority, but a reasonable volume, buzz of 
dissatisfaction." This might be manifested in complaints about overcrowding on the network or 
the inability to perform certain functions or access certain information sources. User 
dissatisfaction then provides the impetus to expand or improve services and support. 

User demand, from both faculty and students, now seems to be based on increasing 
expectations that networking should, and will, be available. Whereas there was a time when 
network administrators had to actively publicize the availability of their services, there was 
agreement among participants that such is no longer the case. "For a long time we were both 
putting in infrastructure and evangelizing the concept of networked services.. ..we don't have to 
evangelize anymore." In fact, the heightened exp)ectations of users have been instrumental in 
causing network providers to accelerate their planning and implementation schedules. This was 
particularly true in the case of incoming freshmen. 

Seeing a whole new crop of freshmen come in, computer literate in a way that we had 
never seen before...all of a sudden this class came in and said. This is our god given right 
and why isn't there a connection in every dorm room?' We had a plan to have that in a year 
and a half, and we...have just sp>ent the last two months wildly coming up with a plan to 
make sure that we got it by September, 1995, because student demand is there. 

In addition to the increased level of demand for network access from on-campus facilities, there is 
a growing demand for access from off campus. 
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Home access is really what did us in in the last six months... it was always something 
hovering there as sort of a problem but it was always contained. Something happened over 
the past six to twelve months where it's just not adequate anymore to have this available to 
people in the office. They absolutely need it now at home, in much bigger numbers and 
much higher bandwidth than they ever did before. 

Thus, growing interest among faculty in collaboration, increasing demands from students for dorm 
access, and from faculty for home access, all are contributing to the development of network 
technologies and services. 

New technologies, including hardware and software, also drive the development of 
networking. The impetus to implement these new technologies comes from both users and 
computing services staff. In part, the demand among users for network access has been driven by 
the availability of information and applications online. For example, "The course and teacher 
evaluations... used to be printed and then they decided to stop printing and put it online. And that 
has worked well. It has also driven demand for access..." 

Also, institutions are faced with keeping up with changes in the industry, changes which 
users — faculty and students — are aware of and want. One participant spoke of faculty awareness 
of client/server computing and how that affected plans for replacing mainframe computers. 

Y ou have a political correctness problem, because in a lot of institutions, buying another 
mainframe or upgrading it is not. .if the institution is going to buy a new mainframe 
[people believe] that there is something... there must be something terribly wrong with the 
administrative group or central computing group. 

However, computing professionals added that they do not rely on users to request new 
technologies or services. As professionals, they feel responsible for monitoring the market and 
selecting appropriate new technologies for their users. 

Y ou definitely talk to people who have work to do and [ask] what they think their work is 
going to be like in a couple of years.. ..but I guess the thing I would be careful about is that 
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if you talk to someone about what they wish they could do and they tell you...l don't know 
how to put this, but it's almost too late. You almost haven't done your job if you can't 
anticipate that from a technical standpoint 

They also select new technologies to replace older ones which are outmoded, cannot support user 
demands, or caimot be maintained efficiently. 

A lot of it is dictated by antiquated systems that are, that have been, outstripped by federal 
regulations. And you are basically staring at the option of taking something that was 
written, you know, fifteen years ago and patched ever since, and having to do a complete 
overhaul...or pitching the whole thing.. ..the stuff was written in four different languages 
and so when you get beyond the ability to hire people who understand... COBOL or PL/1 
or whatever these things were written in you have a hugely labor intensive maintenance 
problem... 

The aging of technologies, as well as the availability of, and demand for, improved technologies, 
combine to drive the development of campus networks. 

Institutional support for networking may also be driving its development. Participants 
agreed that there seems to be an emerging consensus among top administrators, at least at their 
institutions, that incoming students will expect a high level of network capability and support and 
that their institutions must provide this in order to remain competitive. 

In planning for our network, we put together a requirement that all the buildings had to be 
interconnected. And we gave the president an option of putting networks in all the 
dormitories. And he basically came back and said without that option he wouldn't approve 
the plan. He wanted the whole thing.... undergraduate education was the driving force for 
the whole process. 

In addition to providing network capabilities and support for students, there is a recognition 
among university administrators that a network is essential in order to attract good faculty. As one 
participant explained, quoting a university provost, "The world is now very different and every 
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faculty person we recruit needs a dowry, needs to understand what kind of workstation they are 
going to have on their desk, what kind of networking connection." 

In fact, at some institutions, and in some disciplines, network technologies and services 
may no longer be viewed as an added inducement to attract faculty members. Rather, they have 
come to be viewed as minimum requirements. 

I think that five or six, or even more, years ago— and I'd like to say that we were an early 
adopter in networking— it was an attraction. Now it is a god given right. There is an 
expectation that it will be there and it will be available 24 hours a day, seven days a week, 
absolutely reliable, bandwidth unlimited, and at absolutely no direct cost. That's 
expected.. ..To the extent that you do not have what is expected and what is the ideal, it is a 
detriment. I don't think anymore that it is a big attraction. It's an expectation. 

Many faculty now view networking as an essential function on campus. And administrators share 
that view. 

It's like a good library. If you are going to be a major research university, and you want to 
attract and keep the best faculty, you just need to have that infrastructure.. ..This is true in 
the sciences, somewhat true in the social sciences, and now becoming true even in the 
humanities. 

In this changing perception of the infrastructure and the elements which comprise it, networking 
often is considered another utility. 

There is an analogy in telephony. Is there not a god given right to a dial tone? Could we 
expect to recruit a young, hot shot assistant professor and say, " Y ou get a desk, you get a 
part of the secretary, you get lights, but you don't get a telephone?" 

Thus a certain level of network development is perceived to be essential to the functioning of the 
university. In addition, universities may be recognizing that computer literacy and network literacy 
are critical elements of an education designed to prepare students to function in an information 
society. 
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[The administrators] accept the fact that if they don't equip their undergraduate students 
with knowledge of technology that these people will be shortchanged as they go out into 
the world and compete with other students for jobs and careers. It has become so 
ubiquitous a part of the corporate job structure that institutions now recognize that they 
have to provide this sort of infrastructure....! think that's why a lot of presidents are 
willing to make the investment They talk to the people on their board, they talk to 
business people, they talk to students, and they realize that this is the way the world is 
moving, and if they don't move that way too... they 'll be left behind. 

Additionally, participants viewed network technology as essential to preparing students to 
go into a work world where collaboration and cooperation are important. Students are 
"understanding how to put groups together and work together collaboratively over this stuff. 

That's something that's another dimension that I'm sure we're teaching them. But I'm not sure 
how to measure that" 

Participants also identified "the convulsion that is happening in collaborative research and 
collaborative education " as another factor driving the development of networking capabilities. 
Participants described 

...students who are up at four in the morning using the network because the old model, the 
old Socratic model, of one instructor and fifty students is no longer 
operative.. .Fundamentally this is changing the way teaching and research are being done. 
Thereby what we are doing is wholly inadequate in and of itself. 

In smnmzuy, participants viewed the changes in the types and levels of support required by faculty 
and students as contributing factors in the development of network technologies and services. 

Barriers to Network Development . In spite of the increasing demand for networking 
among some groups and growing recognition on the part of administrators of the importance of 
networking, a number of barriers to the growth and development of networking exist. Barriers 
participants identified include: uneven growth in user demand; problems associated with network 
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technologies; and institutional limitations, both financial and pedagogical. 

Participants identified growing user demand as a driver of network development 
However, they also identified the nature of that growth, and their inability to predict timing and 
patterns of growth , as barriers to network development. 

Something that we have heard over and over here, but it is indeed a barrier to us providing, 
in a timely manner, the services we think we should, is the very lumpy demand 
curve.. ..Because what happens is.. .there is a pent up demand that wells up.. .until all of a 
sudden there is this crushing demand. Then there's the giant push.... The resources 
become available.. .but now it's a panic. Now it's in response to this overwhelming pent 
up burst. 

Another negative aspect of the growing awareness among users of networking is the 
problem of misinformation. While users are more knowledgeable about technological 
developments, their understanding can be somewhat shallow. " [The hype] raises a lot of 
expectations and it also raises a lot of fears. A lot of what you read about in the press about 
cyberspace is somewhat negative and causes a lot of interesting.. .knee jerk policy discussions as 
well." 

In spite of this growing user demand, there are still many members of the academic 
co mmuni ty who fear and avoid the technology. 

Anybody who has been at a university for ten years or longer.. .can be assumed to be 
relatively technophobic.... [for] the incumbents, it's a very large training and education 
program that we have to put in place...to essentially pull the bulk of the existing staff up to 
a level of competence where they are actually going to be able to make, on a day to day 
basis, effective use of anything you put on their desk, from a GUI interface for purchasing 
all the way to monitoring news groups on the Internet that are relevant to their particular 
discipline or function.... it takes a lot of work to get people to a level where they can deal 
with this stuff. 
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Providers of network technologies and services must educate and train both the technophobic 
groups and the groups that are clamoring for the latest, although possibly inappropriate, 
technologies. 

But fear is not the only reason people do not take advantage of the technology. 

Deficiencies in the technology itself may be responsible for some low levels of use and lack of 
support for further development. A major challenge, to network managers, is 

...getting our systems to be easy to use. They're still not good enough. They're not like 
dialing a telephone to get what you need.. .the systems are not intuitive and easy to use. 

And there are a trillion different kinds of systems and almost a trillion different interfaces to 
access them...the systems are getting more complex as we put more and more information 
services up. 

And as the technology becomes increasingly complicated, the provision of support and 
maintenance becomes more challenging. "Dealing with the networked environment, where you 
have many places to fix and go wrong, is much more complicated than word processing on 
standalone computers hooked up to a printer. " 

Another aspect of the network technology which stands in the way of development is its 
distributed nature and the problems that creates. 

It's the management of this distributed structure. Anyone can get an Internet address and 
hang a server on the network. And if they don't manage it properly and start any password 
that comes through that server, your whole network is exposed. There are a whole lot of 
issues that the mainframe administrator used to handle. It's now distributed all over the 
network. We used to have a mainframe mail system, and if the system was abused, we 
had people who would go out to talk to the people who abused it and try to educate them in 
the proper use. Now you've got mail systems all over the campus, and who's responsible 
for seeing that system administrators pay attention to these kinds of things? It's a whole 
raft of management issues we took for granted in the mainframe, which we haven't even 
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begun to think about how to do. 

Yet another barrier to the development of networking is th e lag between user needs and the 
availability of the technology t o meet those needs. 

We are trying to figure out how to deliver broadband that integrates voice, data, telemetry, 
video. And I think right now a major problem for us is that we are at a kind of a 
crossroads. We're not there yet. And so you are forced with investment decisions on what 
you believe is older technology, but the new technology is not quite in place yet.... at least 
in terms of commercially available production services or services that you can out into 
production. They are not quite there yet. 

And it's not only the hardware and software, but the information resources as well which 
may lag behind demand. Participants saw themselves as 

...being somewhat at the mercy of what services are available.... we buy just about 
anything that will become available or any network service that is available that is 
appropriate to our collection development policy. We try to twist vendors' arms about 
what's available. But if you look at our offerings in terms of information services, library 
collection kinds of things, it's very much dictated by what's in fact on the market. 

And, on occasions, technology developments, and the sudden awareness and demand for 
new technologies, take computing professionals by surprise. "The emerging technologies are 
moving quickly... .all of a sudden new technology becomes available that puts an incredible.. .it's 
the old story of everybody flushing during half-time during the Super Bowl. We have to cope 
with that" 

Finally, part of responding to the demand for home access, as described above, is what 
participants referred to variously as "the last mile problem," or the problem of "the small pipe to the 
outside world." However, participants believe that they can't solve the problem themselves. The 
solution will require them to work together with groups outside their institutions, like utility 
companies, including power, cable, and satellite providers, as well as vendors of technology. 
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services, and resources. 

Other barriers to the development of networking include the lack of appropriate models for 
teaching in a networked environment, inadequate user support, and limited financial resources. 

While the new network cqjabilities enable new approaches to educational processes, there 
are few models of how to use the capabilities . "Another barrier, then, is pedagogical models that 
take advantage of the technology... people are fundamentally automating old things.. .most of our 
professors haven't really internalized how to use the technology to really change the way they 
conduct their classes." 

In addition to providing new technology, there is the task of providing training and support 
to users of the new technology. 

We're talking here about retooling and retraining the staff as well as users.... There are 
significant management issues as well. Structuring and organizing and that sort of thing. 
Now you suddenly need a world class help desk like they have in big companies, in 
industry, that we never needed before. First class network engineers to deal with 
bottlenecks in traffic...you've got to change your organizational structures to deal with 
this." 

Changing those structures as well as adapting the physical facilities to accommodate the new 
technology require financial resources which many institutions simply don't have. 

We simply can't get the financial resources to solve the problems that [we] understand. We 
have 500 classrooms, and they are averaging 30 years old, and in order for you to teach in 
the classroom using a computer and projection system, you need to have new shades, new 
acoustics, new this, new that, and at an average cost of a couple million dollars per 
classroom, or whatever it is. Pretty straightforward way to solve that problem: it takes 
money, and there is not enough money, and you just have to slog it out. 

This lack of resources to buy the technology, along with a lack of appropriate models to apply the 




technology, represent barriers to the continued development of networking on campuses. 

Measures. A final issue which was considered during the focus groups was that of 
measures of the impacts of networking. Participants discussed financial measures, adequacy 
measures, and impact measures and agreed that while traditional measures of technology impacts 
are often inappropriate, new measures have yet to be developed. 

Participants agreed that financial analyses of the costs and benefits of technology may be 
inappropriate to networking "All of the traditional models, all of the accounting models, just don't 
apply anymore.... [we need] to develop new models, and these models are going to be squishier." 

Approaches to measurement which focus on ROI (Return on Investment) have, so far, 
revealed few financial benefits associated with investment in networking. "The one thing I know 
is that we haven't lowered costs. And some people look to [networking] to lower costs. And I 
think that is probably a holy grail that I don't know how to get to." 

However, participants also agreed that they experienced very little pressure to show cost 
reductions and financial returns on investment. There was some suggestion that academic 
administrators do not ask for economic justification of requests for investment in new 
technologies, or in other improvements to the institution. 

Colleges and universities don't make these decisions based on direct economic issues. 

They don't ask what is the cost benefit.... they don't ask that with respect to anything they 
do. I mean there is no bottom line, there isn't.. ..in a corporation there is a balance sheet 
with the technology input and the labor input and the cost of the product and its 
competitiveness are all evaluated. And whether you can afford to upgrade the technology is 
a matter that is determined by whether you can pass the cost off to the product that you are 
producing. That never happens in higher education.... It's totally not an economic model. 
Although, for capital dollars, there were instances where economic justification was 
required. 

Nobody is throwing capital dollars at networking [at our institution]... they are expecting to 
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see some sort of supporting improvement, or some sort of recovery scheme to come back 
to recover the capital costs. It's a business improvement.. ..In the case of undergraduate 
access, we basically advocated direct recovery from the students... assessing a technology 
fee.. .to defray a combination of the capital and operating costs. 

In other instances university administrators were reluctant to assess fees from students or 
faculty to cover the cost of network access. 

We have this plan of scaling up home access through a recharge plan.. .and create a new 
service just for faculty.. .and the vice chcuicellor bought it cuid he said, "But I just don't 
want to charge the faculty." It was very much that issue of if something is this important to 
this constituency, we're going to find a way to do this. 

Likewise, another university attempted to provide network access to dorms without 
charging students. "Because of student demand, and the surge of interest in student services.. .the 
university budget office is sitting arormd right now figuring out how to reallocate capital funding 
for this, because this is just so important." 

Often there are financial tradeoffs that have to be made in order to support expanded 
network services in institutions which do not have unlimited financial resources. 

Because we're certainly not immune from all the cost cutting measures going on throughout 
the rest of the institution... we've got to make arguments and internal kinds of decisions on 
trading some things off...but I've never been asked to justify the marginal value of a 
particular set of ports in a place on economic terms. 

Computing administrators sometimes justify an investment to improve networking 
capabilities by demonstrating that money can be saved by the investment. 

The other thing that was demonstrated when the university first put in its Internet 
coimection was that instead of having to buy a supercomputer in order for the physics 
department to be able to do what was considered cutting edge work, all they needed was a 
56 kilobit line to [another university] . And that made them competitive in the grant area. 
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So it's arguments like that that could carry substantial weight, when you can show the 
defraying of some major in vestment.... because they could make use of supercomputer 
centers elsewhere... 

Another facet of measurement has to do with whether the network is adequate to meet the 
needs of users. One measure of this adequacy is universal connectivity. 

Essentially every member of the university community has the ability, the physical, 
technical ability, to connect to the networked resources of the institution.. ..the university 
has just invested several million dollars to bring a backbone network to every building in 
the institution with the object of ....having every workstation connected to the backbone 
network.. ..As well, the university just finished investing in putting an Ethernet port to 
every student in the dormitories.. ..So every room has one port per student in that 
room... .the community, the students, faculty, staff, administrative community has the 
technical access...! would say that is an adequate environment. 

Participants suggested several quantitative measures of adequacy. "Certainly one measure 
of adequacy is how many desks you reach.... Another measure is certainly what bandwidth and 
what capacity you are giving them." And there was agreement that there are some basic 
technological requirements that would have to be included in any definition of adequacy. "Today, 
because of high bandwidth requirements of multimedia, an institution has to have, first of all, an 
adequate LAN infrastructure with adequate bandwidth." 

But the definition of adequacy may not necessarily extend to actual use of the technology 
and services. Whether departments or other units in the university choose to take advantage of the 
technology is another issue. "The urgency with which a given department makes the investment to 
make the connection can vary from one discipline or function to another." 

Another measure of the adequacy of the networked environment is related to support of the 
technology described above. 

The second important element is support infrastructure, the size of the networking group 
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and the user support for all of the problems that people have. If the support organizations 
don't have the manpower or the knowledge or whatever, then you are going to have 
problems. 

Also, the ability of individual departments to manage their own networked information 
resources was considered an indicator of adequacy. "Adequately preparing departments to build 
their own and mount their own information... we can at least count that and judge whether that is 
adequate in some sense. " 

Another facet of measurement has to do with the information resources available via the 
technology. "It's not just the wires into the place, it's certainly what resources they get.. ..The most 
common network resources are the library, most institutions that have a network have coimected 
the library.. .and a large number of institutions have... campus- wide information." 

Beyond the measures described above there was agreement that there is another, broader 
measure of the network's capacity to meet users' needs. 

It's not sufficient to talk about bandwidth and connectivity....you've really got to talk about 
all the services and capabilities which reside on the network. It seems to me, if you are 
asking questions of adequacy, you have to ask the questions about are the services which 
assist the individual in doing whatever it is that they are going to do. 

Moving beyond financial measures and adequacy measures, participants discussed a developing 
interest in measures of the impacts of networking on teaching and learning. Administrators are 
beginning to ask questions like, "Has it enabled an instructor to increase the contents or broaden 
the contents or get deeper in the content of the class?" and "Has it reduced their administrative 
work in administering the class?" 

In response to such questions, network administrators are beginning to develop and apply 
new measures. For example, "We provided multimedia support for a classroom [and] we have 
evidence that the faculty member [using the classroom] is spending less time on the mechanics and 
more on the content So that the students are getting more content and learning it faster." In a 
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similar case, another participant commented on faculty members spending less time writing on the 
blackboard, or photocopying and distributing materials to students, when the information was 
made available over the network. 

Another participant reported on a study done to determine if students believed that their use 
of electronic information services affected their work or their grades. "Again, you know, it's still 
soft in the sense that they me asking users their view, not that there's an external quantitative 
measure. But there is an effort at that kind of measurement." However, most evidence of 
networking's impacts on teaching and learning is anecdotal. 

It's a faculty member saying, "We've got an architecture class, and we're doing shared 
design projects with students in Norway." Another faculty member saying "I have a small 
class and there's another fellow in Nebraska with a small class, and we are collaborating 
using the Internet." Another faculty member is saying, "I tried some things with computer 
mediated communication to get everybody to join the discussion and I tell them that their 
grade will be based on their level of participation in an online discussion. I have more 
consistent participation by every student in the class than I ever had in a real classroom." 
There are all kinds of things like that that you can point to that you can say that those are 
things that could not have happened any other way. So those are tangible outcomes but 
you can't measure them. You can't say, "Did they get a better grade?" "Did it attract a 
whole new class of clients?" It's hard to quantify. 

Another problem identified in trying to create measures of the impacts of networking on 
activities like teaching and learning is the lack of good measures of these activities, regardless of 
networking, and the lack of existing data on teaching and learning in a non networked environment 
at some institutions. As one participant cautioned, 

I think that a major problem you are going to run into early on is that we never really 
measured these outcome measures or evaluated the quality of instruction or learning or 
anything anyway. So now we are asking how has this proved something that we never 
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measured anyway. 

Participants also reported on their efforts to develop measures of networking's impact on 
research. One person described a study in which he compared the number and value of research 
grants received by faculty who were active users of the network to those received by faculty who 
were not active network users. 

There was an amazing correlation.. ..between those that had the network connection and 
substantially higher amounts of research dollars that came in.. ..I did it within humamties, I 
did it within hard sciences and did it within engineering, did it within medicine...it was 
there, it was indisputably there. [But] there are probably a lot of intervening variables." 
Other measures of networking's impact on administrative processes also are emerging. 

The largest dividend that I think you'll ultimately be able to point to is some sort of 
reduction in paperwork. And some sort of reduction in lead times, to be able to reengineer 
processes and adjust the university's work to some new, speedier way of doing work 
electronically. 

But the impact of networking goes beyond reducing effort and speeding up work 
processes. "It's not just improving what we're doing. It is totally enabling us to do different 
things that we couldn't do before.. ..so it's not just doing what we have done and doing it faster 
and better and more efficiently. " For example, networking may be a factor in changing the market 
for higher education. 

I think in this next (state) legislative session... we will see the legislature require us to 
permit high school students to have simultaneous registration in college courses in high 
school. And we couldn't do it without the network. I mean the legislature couldn't even 
think of that. 

Although there are examples of the impacts of networking, as described above, 
measurement of these impacts remains very difficult. As one participant described the situation, 
“We are at a very immature stage where we really, I think, are only getting glimpses of what the 
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future is going to hold. And so, it's going to be very hard. I'd say, to measure things, because it's 
very foggy just where all this is going to go.” 

The development of networks at academic institutions is in a period of rapid growth, fueled 
by user demand, the emergence of new technologies, and institutional support. There seems to be 
little doubt that networking is becoming crucial to student learning, faculty teaching and research, 
and institutional survival. 

However, barriers to the continued development of networking remain. It is difficult for 
academic computing departments to anticipate and support user demands. And the technologies, 
services, and information resources users want and need are not always available and, when they 
are available, they're not always easy to access and use. 

In spite of these barriers, there is evidence that networks are having impacts on people who 
use them and on the institutions in which they work. Quantitative, empirical evidence of these 
impacts is rare. Yet there is anecdotal evidence as well as an intuitive feeling among academic 
computing professionals, that access to network technologies and services is changing and 
improving the ways in which academic processes are conducted. 

A final issue which focus group participants considered was the development of a model of 
the academic networked environment. They agreed that there are many elements of the 
environment which are consistent across institutions. "If you look at this in detail. ..it is 
remarkably consistent across institutions.. ..people [aren't] creating different academic networks." 
"There is a real variation in the funding [across institutions], and yet physically those networks 
look virtually the same." 

In addition to similarities in the physical structures of networks, there appears to be a group 
of services which are also common to many institutions. "Everyone has electronic mail. Everyone 
has a directory. Everyone has a campus- wide information system. Everyone has listservs." 

Another element of the academic networked environment model is "the constituencies that 
make up the academic institution.... administrators, staff, faculty, students, and the community." 
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And a final element is "the support structure... what's traditionally been called user 
services... helping people along, helping them grow and mature in this kind of environment, 
enabling them to make full use of the resources." 

The purpose of conducting these two focus groups at this point in the investigation was to 
inform the design and implementation of the planned case studies. Specifically, we sought to 
ground our future data collection efforts in an understanding of the current state of networking at 
academic institutions based on empirical evidence. 

The results of these two focus groups provided evidence to support some of the 
assumptions on which this study is based yet seem to indicate that some other prior assumptions 
were incorrect. Also, the focus groups identified a number of issues which had not been 
considered in the original design of the study. The following conclusions were drawn from these 
focus groups: 

• Clearly, the growth of, and interest in, networking at academic institutions are increasing 
rapidly. Networking is becoming an essential part of academic institutions. 

• There is interest in measures of extensiveness, as related to networking, and academic 
computing professionals regularly use a number of simple measures of the range and 
capacity of their systems. 

• There is not a great deal of interest in the ^plication of traditional financial models to the 
evaluation of networks. Focus group participants seemed to agree that these traditional 
models are not useful and they are under little pressure to provide economic justification for 
networks. They also agreed that there is little evidence to support a justification of 
networks on the basis of economics: they don't believe that networks have lowered 
institutional costs. 

• There does not appear to be much of a challenge on the part of academic administrators to 
the idea of networking. Instead, there is much agreement about the necessity and value of 
networking. 
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• Networking has come to be considered a utility by users and providers. It is no longer 
something which is optional. 

• There may be a certain minimum level of network capability required to attract and retain 
both faculty and students. 

• The real value of networking is in what kinds of activities it enables which could not have 
been supported at all in a non networked environment. 

• The full advantage of networking may not be reali2ed until new pedagogical models are 
developed. Currently, many existing educational processes are being automated, but those 
processes are not being redesigned to take full advantage of network technology. 

• Networks are becoming increasingly complex and theref ore more difficult to support and 
maintain. 

• Networks are becoming increasingly difficult for academic computing services to manage 
because of their distributed nature. As users become more sophisticated, technologically, 
they are able to assume more responsibility for the management of their own network 
technologies and services. 

• Existing technologies are lagging behind user demand. 

• The availability of information resources in the networked environment is lagging behind 
user demand. 

• As demands for network technologies and services grow, and academic budgets shrink, 
investment in, and support of, other resources and services within the institutions may have 
to be cut in order to support networking. 

• Evidence of the impacts of networking on academic institutions is primarily anecdotal at 
this point. 

• The absence of good measures of teaching, research, and learning prior to networking will 
make it difficult to assess networking's impacts. 

It appears that there is an almost universal acceptance of the importance of networking in academic 
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institutions, and little pressure to identify financial benefits associated with the costs of developing 
and maintaining network technologies and services. However there is a growing interest in 
developing methods to assist users in taking full advantage of the new technologies and in 
providing information resources which users can access via the networks. 

4.2.3. Demographic Survey of Focus Group Participants 

Following are summaries responses given by the university computing focus group 
participants and the CNI focus group/small-group interview participants to the demographic survey 
questions. 

UNIVERSITY COMPUTING : 

Professional Experience . Table 4-1 shows that most of the participants were mid-level 
managers with limited ACS experience. Four participants worked in ACS for 3 years or less, and 
only at their current institution. The other three participants had more than 10 years of experience 
in ACS. This group had technical expertise, but their perspective of the academic networked 
environment lacked the comprehensive overview of upper-level managers. 

Table 4-1. Professional Experience. 



Type of Position 


Length of T ime in 
Current Position 


Length of Time in 
ACS 


Director 2 


0.25 - 3.5 


10- 16 


Sr. Consultant 1 


2 


19 


Manager 3 


1.3-3 


1.3-3 


Programmer 1 


1.25 


1.25 



Rest>onsibilities and Expertise. Four of the respondents were responsible for the technical 
aspects of the network, and two others provided service support (e.g., instructional technology 
services and helping the faculty with all their computing needs). The four participants who 
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managed the network listed the following responsibilities: 

• Coordinate development, deployment, and support for central network services 

• Manage network infrastructure and Internet connection 

• Manage network design, support and troubleshooting 

• Manage the CMS provided Novell servers and AUX services. 

Whether managing the system or providing support services, all the participants dealt with the 
network and information infrastructure as a primary responsibility. Participants listed four general 
areas of expertise : 

• Network infrastructure 

• Operating systems 

• Applications 

• Management. 

Six participants listed some aspect of the network infrastructure as an area of expertise. Some of 
these areas included ethemet, TCP/IP, LAN and the Internet. Four participants listed operating 
systems, such as UNIX and desktop operating systems, as their areas of expertise. Two 
participants listed expertise with the following applications: 

• Interactive video 

• Classroom presentation 

• Instructional design 

• Social science applications. 

Only two participants listed management as an area of expertise. 

Evaluations. There was no consensus with respect to formal evaluations as most of the 
participants were unable (or unwilling) to answer these questions. This was more likely a 
reflection of their position within computing services. Two participants said that user studies are 
conducted as part of IT planning and evaluation, the others either didn't answer or answered no. If 
user studies were conducted, it does not seem that the data were distributed to all computing 




service personnel. In response to a question about their personal evaluation of the school’s 
information infrastructure, six participants rated their institution as "Slightly Better" than peer 
institutions and one rated it as "Much Better." 

The participants listed the following criteria for assessing their institution's information 
infrastructure: good infrastructure (described as wiring and wiring plant, distribution of high-speed 
lines, and network design), reliability, performance, service, access and good support personnel. 

Challenges. Participants identified the following challenges to providing a good 
infrastructure: training (of support persormel, end users, and “the computer illiterate.”); and 
technical challenges (such as staying up-to-date and making the right choices for the netware 
software). 

CNI : 

Professional Experience . All but two of the participants in the CNI focus group/small- 
group interview held major positions within their institutions’ academic computing services 
departments. One participant had left an academic institution to become the director of a 
consortium, and another was a university librarian. Ten participants each had at least 10 years of 
experience in academic computing, one participant had 40 years of experience, and all of the 
participants had major positions within the academic computing services hierarchy (see table 4-2). 

Table 4-2. Professional Experience. 



Current Position 


Length of Time in 
Current Position 


Length of T ime in 
ACS 


Director 


4 


2-11 


10-40 


Deputy Director 


7 


1 - 14 


2.5 - 22 


Manager 


1 


0.25 


7 



Responsibilities and Expertise. Eight participants were responsible for supporting, 
coordinating and managing their Campus-Wide Information System (CWIS) and information 
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technology. The CWIS includes: desktop computing, Internet access, public computer clusters 
and academic/administrative and telecommunications computing. Only one of the participants 
listed education of end users as a job responsibility. 

Half of the participants listed management as an area of expertise; nine listed computer 
networks and networking as areas of expertise. Strategic planning/vision was an area of expertise 
for three of the participants. 

Evaluations. Six participants said that their institutions had conducted evaluations of the 
network and computing services within the past three years. (Two of these six participants were 
employed by the same institution.) 

Participants said they used informal, rather than formal, evaluations to compare their 
institutions to others. Table 4-3 compares the informal rating of the infrastructure by institutions 
with regularly conducted evaluations to those institutions which do not conduct evaluations. 
Perceptions of the quality of the infrastructure were similar regardless of formal evaluations being 
conducted. 

Table 4-3. Quality of Infrastructure. 



Evaluation Network? 


Quality of Infrastructure 


Yes 


Much Better - 2 
Slightly Better - 3 
Same - 1 

Slightly Worse - 0 
Much Worse - 0 


No 


Much Better - 1 
Slightly Better - 3 
Same - 1 

Slightly Worse - 0 
Much Worse - 0 



Participants considered the infrastructure and universal access to the network as the most 
important criteria used to assess the overall quality of their institutions’ information infrastructure. 
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When discussing the infrastructure, they listed the following criteria: 

• Penetration and diffusion of departmental computing 

• State-of-the-art technology 

• Ubiquitous connectivity 

• Reliable performance 

• Robust networking 

• Extent and capacity of the network. 

Services such as end-user support and training as well as the nature and array of services 
were also considered when evaluating their institutions. One participant whose institution 
evaluates the network stated that a survey of user satisfaction was one of the criteria used in 
judging his infrastructure as "Much Better" than peer institutions. 

The two participants who rated their institutions "The Same" as peer institutions used the 
following criteria: 

• Access to administrative information 

• Universal access 

• Degree of end-user support/training 

• Universal service 

• Reliability 

• Cost-effectiveness. 

Unique Services. In response to the question of unique network services, participants 
offered the following: 

• Client software, Internet, Gopher, Popmail 

• On-line library resources 

• E-mail 

• Bitnet information services 

• Network ID and password 
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Access to institutional data 
K-12 Access. 

Museum informatics project 
Meteorological data 
Genetic databases. 

Challenges. The challenges identified by participants were primarily in the areas of user 
services and technology. Participants listed: 

• End-user training to take advantage of investments 

• Curriculum change to take advantage of the investment 

• Ability to provide home-access to full constituency 

• Collaborative effort for teaching and learning 

• Education/use of networks by faculty 

• User training and assistance 

• Ease of use of systems. 

Technology challenges identified included: 

• Sufficient bandwidth 

• Wireless frequency allocation 

• Networking disparate systems 

• Organizing information on the network 

• Seamless integration. 

Other challenges included: 

• Security of information balanced against access 

• Intellectual property rights issues 

• Freedom of speech issues and censorship 

• Ability to develop new policies for electronic resources. 

When comparing the demographic data from the university group and the CNl groups, it is 
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important to remember the differences between the participants' positions within academic 
computing. The university group was composed of "hands-on" middle management personnel and 
the CNI focus group and small-group interview were comprised of upper-level managers. In spite 
of these differences, some patterns emerged. 

Both groups considered information technology, the infrastructure, and the network as the 
backbone of the networked environment. There were no clear definitions of this terminology. The 
participants used all these phrases interchangeably. This lack of standardization of definitions 
made it difficult to accurately assess some responses. User services, accessibility, user training 
and support were also considered important parts of the academic network. 

Because the methodology used in this approach was an evolving one, focus group results 
helped identify questions and issues to be explored during the site visits. For example, it was not 
clear from this survey why some institutions do not evaluate their networks. Is the network too 
distributed? Is it growing too rapidly? Is the technology changing so quickly that studies would 
be out of date before they were completed? Are the institutions so busy trying to build a system 
that they cannot stop and evaluate it? These questions were addressed later, during the site visits. 

4.2.4. CNI Feedback 

The data collected in the initial focus groups were then used to develop a preliminary model 
of the academic networked environment and a trial set of measures to assess the academic network. 
Members of the study team presented this model and the trial measures in a project briefing session 
open to attendees at the 1995 Spring Meeting of CNI. Approximately 50 people attended. 
Following the presentation, the study team members conducted a group discussion of the model 
and measures. The objectives here were to refine the model and measures, collect data on 
assessment activities at other academic institutions, and identify possible host institutions for site 
visits. Following is a summary of the group discussion: 
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• Critique of the model . Participants suggested that the preliminary mcxiel be revised 
to show linkages among the various components, especially between users and 
applications. 

• Critique of the measures . Participants suggested a different categorization scheme 
for the measures: institutional measures, service measures, and individual 
measures. Also, they suggested that there be a stronger emphasis on measures of 
quality and satisfaction. 

• Anticipated difficulties . Participants acknowledged the importance and potential 
usefulness of the measures, but at the same time expressed concern about problems 
associated with collection and analysis of assessment data Specific problems they 
identified were: difficulties in generalizing from the data; reluctance of the parties 
involved to release such data; and the lack of resources available for data collection 
and analysis activities. 

• Suggestions . Participants suggested that the study team contact accrediting 
agencies which have had experience with assessment efforts. Also, representatives 
from two academic institutions which has network and computing assessment 
experience volunteered to provide additional information on their efforts. 

As a result of this group discussion, the study team made a munber of changes in the focus 
and design of the study and in the assumptions on which the study was originally based. Most 
importantly, we expanded our focus to include performance measures as well as impact measures. 
And we also modified our expectations concerning the current state of network assessment; clearly 
less was being done in actual practice than we had believed. This information, along with 
additional information from follow-up contacts with participants, was used to modify the model 
and measures and to select locations for site visits. 
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4.3. Site Visits 

The study team selected and visited two academic institutions for the primary puipose of 
testing and refining the draft performance measures and the Academic Networked Environment 
(ANE) model. Members of these institutions assessed the measures in terms of their feasibility, 
usefulness, cost, and accuracy. The study team compared the ANE model with the existing 
network in each site visited. Thus the site visits informed the continued development of the 
performance measures and the ANE model. 

A secondary purpose was to develop detailed descriptions of the academic networked 
environment at the selected institutions. These descriptions serve as examples of the types of 
network configurations that are possible in academic institutions. Additionally, the site visits were 
used to gather more in-depth information about previously identified issues and to help identify 
new issues. 

The site visit is an investigation of phenomena in a real life context. As such it is a variant 
of the case study method. However, unlike the case study, the site visit typically consists of a 
single visit, is somewhat informal, and the focus is on current, rather than historical, events (Yin, 
1994). Sites are selected according to criteria relevant to the particular investigation. Data may be 
collected through a variety of methods; typically, a combination of methods is used. Analysis is 
primarily qualitative, but may be quantitative as well . 

The study team selected potential sites from those institutions which constitute the larger 
sample for this investigation: those colleges and universities which are classified by the Carnegie 
Foundation for the Advancement of Teaching as a Research University I or II and which are 
members of ARL. The following additional criteria were used to select two sites from that group 
of institutions: 

• the institution was within reasonable proximity of Syracuse, 

• the institution has demonstrated advanced development and 
implementation of network technologies and services. 
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individuals at the institution were interested in the research 



project, 

and these individuals were willing and able to participate in the 



site visits. 



Also, there was a need to contrast a private institution with a public one. Thus, one private and 
one public institution, both of which met all of the criteria above, were selected. 

Within the selected sites, study team members met with and interviewed numerous 
individuals and groups involved in networking on the campus. This included: university 
administrators; directors and staff of computing services departments; library directors and staff, 
esp>ecially those involved in the electronic delivery of library access and materials; faculty involved 
in network use, policy and planning; and students. 

Using a theoretical sampling approach (Glaser & Strauss, 1967), some individuals were 
selected before the visits as the people most likely to be involved in networking. When 
interviewed, those people identified other individuals who were also involved in networking and 
these other individuals were then interviewed as well. Details on the numbers and types of 
individuals and groups interviewed are presented in later sections on each site visit. 

Data were collected during the site visits via interviews with individuals and small groups, 
tours of facilities, observations of services and staff interactions, and collection of relevant 
documents. Study team members taped some interviews, and took notes on all interviews and 
observations. Interviews generally took an hour to an hour and one-half to complete. 

Interviewees were asked to complete a short demographic survey of their background and expertise 
related to networking and evaluation. Additionally, the study team collected data from casual 
conversations while at the sites and through telephone and email contacts following the visits. 

The following questions were used to loosely structure the interviews: 

• What is the current state of networking and evaluation of network performance at 



this institution? 
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• What key issues currently affect successful evaluation of networked information 
services? 

• What are the respective roles of computing services, libraries, vendors, and users in 
the evaluation process? 

• How would interviewees assess the usefulness of the performance measures 
proposed by the study team? 

Each member of the study team who participated in a site visit performed a content analysis 
of her or his notes and, where available, the tapes from the interviews. Study team members 
combined their individual analyses of the interviews with their observations made during the site 
visits and with additional information obtained from relevant documents collected at the sites. 
Analysis of the data was guided by the questions listed above. The individual analyses were 
combined into a single document and reviewed by individual members who participated in the 
visits. Thus, the final analysis is based on multiple perspectives of the site. The interviewees 
raised topics and issues over a broad area but only those responses and views that bear directly on 
project topics are reported here. 

4.3.1. Private University Site Visit 

Three members of the study team interviewed five individuals involved in networking at the 
selected private institution including representatives from the libraries, networking group, 
instructional services, and support services. 

Results of the interviews are organized here according to the following categories: general 
issues related to the development of the manual and project topics, the current status of networking 
evaluation on campus and attitudes toward evaluation, and reactions to the performance measures 
manual distributed to the interviewees prior to the interviews. 
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Interviewees raised a number of issues regarding evaluation of network activities and 
services. To a large degree, they raised these issues indirectly, as part of the interviews, and not 
by specifically noting that the study team should “really pay attention” to a particular issue. 

Focus of the measures . The primary focus of this project has been on academic 
computing. However, many of the measures proposed may apply to all computing on campus. 

For example, when measuring the flow of network traffic over the network it may not be possible 
to parse out academic computing from administrative computing. Measures like the number of 
email messages which travel across the network will include email from academic groups, 
including faculty and students, as well as administrative groups, like deans' and registrars' offices. 
Both academic and administrative computing are encompassed in such a measure. 

Intended audience for the manual . The original intention was to direct the manual towards 
computing administrators and centralized computing services only, in view of the problems 
associated with trying to measure the performance of distributed units which are not under central 
control. However, the interviewees stressed the importance of the distributed units at their 
institution and said that to ignore these units would present a distorted view of the computing 
environment They suggested that the study team enlarge the focus of the manual to make it more 
useful for those in distributed units. 

Central computing measures versus departmental measures . Related to the above issue, it 
became obvious from one interviewee that there is a need for, and an interest in, these measures 
among the distributed units. This particular individual had much interest in developing measures 
that would be useful specifically in one college. However, not all measures will be appropriate for 
all departments, and comparing across institutions and across departments might be problematic. 
Nonetheless, this individual indicated the need for measures that he could use in his particular 
setting and noted that, at this institution, different academic units counted networking activities 
differently and it was unclear if central computing could “force” agreement on how to define and 
count certain network activities/services. 
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Distributed versus decentralized networking . A distinction was made between the 
distributed environment and the decentralized environment. The distributed environment is one in 
which technologies, services, and support are distributed across the campus, but responsibility for 
the technologies, services, and support is centralized. In a decentralized environment technologies, 
services, and support, as well as responsibility for them, are distributed across campus. 
Interviewees urged that the manual recognize this distinction and relate it to evaluation. 

Production of monthly statistical reports on networking activity . Apparently there are a 
number of units on campus that produce regular statistics. A monthly report on traffic is produced 
from central computing services. Unfortunately, members of the study team were unable to meet 
with individuals responsible for this activity. Additional information about this process as well as 
example reports were obtained later. 

Understanding relationships among txjssible measures . Interviewees were especially 
interested in having measures that might assist them in determining how resource allocation for one 
type of activity might affect another activity. For example, one person commented that additional 
resources being committed to the help desk might be “throwing money into a dark and bottomless 
pit.” Thus, if the university increased expenditures for network training and instruction, would 
reduced expenditures be needed for the help desk? Interviewees were interested in the degree to 
which the measurement manual would address such relationships. 

Network sniffers and probes for evaluation . When asked about the extent to which central 
computing could measure traffic or other aspects of network activities, the response was, “When a 
problem occurs, we attach a sniffer to that particular router or server to find out what is 
happening.” The term “sniffer” refers to hardware/software that network management would 
connect to a system component where a problem occurred to “sniff’ out or probe the problem. But 
the cost of attaching sniffers to the network is significant, and could only be done on a “sampling 
basis for short periods of time for purposes of evaluation. 
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The selected private institution has significant networking resources in place and is 
planning for its next network configuration which will be ATM (Asynchronous Transfer Mode) to 
the desktop. Within this context, however, there are numerous issues and problems that the 
university faces, as summarized below. 

Evaluation of netwoik: activities and services is a “good thing .” In general, all of the people 
interviewed agreed with this view. But they also agreed that none of them conducts evaluations 
regularly. There was an underlying assumption that the network is a good thing and that the need 
for it is essential and growing. Therefore, the interviewees explained, evaluations to determine 
what’s wrong with the network, or whether the network is necessary, are not needed. As one 
interviewee said, “It’s not like I’m going to come out with an earth-shattering study that’s going to 
prove tec hn ology is worthless and we’re all going to go back to books. It’s not going to happen.” 
Inequalities of computing resources. A network administrator described the university as a 
very heterogeneous environment, where 

“...some college units are relatively resource rich and some are relatively resource poor and 
it's got more to do with the historical situation than with anything that's evolved because of 
the structural needs of the information technology. Addressing that imbalance is going to 
be one of the inunediate items on the agenda over the next few years, addressing it in some 
formal, systematic way.. .and addressing where the line is between central and distributed 
support." 

He recognized that having some type of performance measures could assist them in dealing with 
this issue, and, over time, determine the degree to which “progress” in equalizing resources had 
been made. 

Non-svstematic collection of networking data. There was evidence that some data on 
network performance are being collected, by different units and by different people. However, 
there is little evidence that the data are being collected and analyzed in a systematic way, or that 
they are being used in planning and decision making. Interviewees agreed that having a central 
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management information system (MIS) that identified, collected, organized, analyzed, and reported 
selected networking statistics would be an important step forward and was essential for improved 
pl anni ng for networking services. They also noted a range of problems and issues that would have 
to be resolved if such an MIS were to be established at this particular institution. 

Rpanhing agreement of networking terms and evaluation procedures . An interesting 
discussion took place regarding the use of different procedures and different terms for the 
evaluation that was currently being done. There was little confidence that central computing 
services could obtain agreement from other stakeholders on campus as to the appropriate 
definitions to be used in a larger institutional effort on evaluation. One person commented, “it 
would take an act of God to reach such agreement.” 

Overview of barriers to ongoing networking evaluation . Several reasons were given for 
the lack of systematic data collection and low priority placed on evaluation activities. 

• Some interviewees were suspicious of the goals of evaluation. When asked what 
his response would be if his director asked for this typ>e of data, one person said, 
"My first question would be, 'What are you going to do with it?'” 

• There are "power pockets" throughout the university and a grossly unequal 
distribution of resources. Thus, some groups have a vested interest in not sharing 
information about their resources lest they be pressured to share those resources. 

• Individuals charged with providing network access to a growing, and an 
increasingly demanding, group of users may not have the time and resources to 
conduct evaluations. Their focus is on keeping the network rurming and meeting 
users’ demands for speed and power. As one administrator described it, "Life is 
very simple for me as a network planner. I need to keep figuring out how to get the 
best, biggest, fastest, cheapest pipe in here possible on the presumption that the 
need for bandwidth is going to get bigger and bigger and bigger..." 
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• The lack of support (e.g., additional resources, a graduate assistant to do data 
collection, etc.) to conduct such evaluations and for some, limited knowledge on 
how to conduct such evaluations is also a barrier. One person indicated that before 
he would feel “comfortable” doing such evaluations, additional training would be 
needed. 

• The lack of incentives to conduct such evaluations. While generally agreeing that 
evaluation was a good thing, they also noted that there were few tangible and direct 
incentives for conducting such evaluations. As one person commented, there was 
not an institutional mindset supporting ongoing evaluations. 

• The oftentimes confusing distribution of technologies and services, and 
responsibilities for managing those technologies and services, compound the 
problems associated with evaluation: it is not always clear who is responsible for 
what. A number of the participants commented that they were unsure who did what 
in terms of networking or were unsure who should be contacted to solve a 
particular networking problem. 

• There are considerable difficulties in defining key networking terms and services in 
such a way that they can be operationalized for measurement. Thus, before 
evaluation can occur, the institution may first have to reach agreement on how to 
operationalize key networking activities for measurement. 

• The recent rapid growth and change in networking makes evaluation and planning 
extremely difficult. It is not always possible to predict the next direction in the 
tec hn ology “I remember when someone said, ‘Who needs a laserwriter? What are 
you going to do with that?’ And desktop publishing. Nobody could have predicted 
this stuff.” 

• Networking infrastructures, services, and administrative orgamzation for 
networking change r^idly, making evaluation difficult in such a volatile 
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environment. For example, on the day that the study team conducted the 
interviews, the Vice President for Computing announced a re-organization of the 
management responsibilities for computing services on campus. 

In summary, interviewees agreed they would evaluate network performance if: 

• someone ordered them to do so, 

• they believed that the results would bring them additional 
resources, 

• they could expect to receive additional personnel to conduct 
evaluations, 

• they had training in evaluation methods, 

• and they had the time to conduct evaluations. 

In preparation for the visit, the study team supplied interviewees with a description of six 
proposed measures from the draft performance measures manual. The study team was pleased to 
note that most of the interviewees had read these proposed measures and some had made a number 
of margin notes as a basis for their discussions with the team. Following is a summary of the 
reaction to the manual and measures. 

Identifying issues that require institution-st?ecific restxjnses . The approach taken in the 
.manual, which presents issues associated with data collection and analysis and then leaves it up to 
individuals to resolve those issues in the context of their own organizations, was generally 
accepted. Interviewees recognized that it would be impossible to propose data collection 
techniques that could be used in a range of different institutions. They also agreed that before any 
type of data collection could take place, the institution may need to develop a range of policies and 
guidelines for how the data will be collected. Those policies would have to come from the 
institution and not from the manual. 

Receptiveness to the ideas of measurement and to measures the study team proposed. 
Interviewees generally liked the measures that were presented although they also wanted more 
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detail on procedures. One person commented that if he had such a manual now, it would greatly 
assist him in conducting such evaluations. 

Inclusion of anecdotal information and assessments . Interviewees urged that the study 
tpam not disregard anecdotal evidence of the impacts of networking. The network administrators 
interviewed said that they have used anecdotal evidence to help support their requests for resources 
to improve their networks and they themselves value such evidence. They also agreed that they 
need training in the methodologies used to collect, analyze, and present such evidence. A section 
in the manual that discussed techniques to systematically collect, organize, and report anecdotal 
information could be very useful. 

Use of focus groups rather than, or in addition to. a user survey . One person said that 
instruction in techniques to obtain assessments from users other than surveys, such as focus 
groups, would be helpful. He believed that user surveys could require more time than available, 
whereas conducting focus group sessions with selected types of users could yield equally useful 
information while requiring less time. 

More emphasis on impacts and effectiveness . One administrator described the measures in 
the manual as "...a good start at getting some reasonable measures that I would use in a tactical 
way to make decisions about capacity planning and that kind of thing." Y et, he said the measures 
fell short of conveying the true impact of networking on campus: 

“The thing that’s really most liberating about the technology is the fact that it frees you from 
the constraints of time and space.... it means that there are certain characteristics associated 
with it that have the possibility of enabling you as an information processing organism to 
do it differently than you’ve done it before. Look at things like feedback loops. It’s 
possible to use the technology to get a quicker feedback loop. What does that boil down to 
in practical terms? The fact that I can access with very little effort all kinds of different 
opinions and perspectives on an issue. What I find myself wondering about is what is the 
effect of that developmentally, if you were to grow up with that kind of stuff.” 
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Others commented, especially in the instructional arena, that measures showing use and knowledge 
of networks positively affected learning would be essential and that they needed such measures 
now\ 

Another individual suggested that the study team review the assessment areas in the 
CAUSE “Self-Assessment for Campus Information Technology Services” (1994) by Heit to 
identify additional areas for possible impacts assessment. 

Intended level of analysis of the measures . Several of the people interviewed made 
comments and asked questions which indicated that the level of analysis at which this research is 
being conducted is not clear. The measures proposed are, primarily, at the institutional level. That 
is, they are measures of network activities and expenditures for the entire institution. While there 
are effects of networking at the individual and group levels, the measures proposed are not 
intended to c^ture those effects. However, the measures proposed will form a foundation on 
which to build measures of effects at the individual and group levels. 

Expanded introduction . Although not directly discussed by the participants, it was clear 
that any manual would need to have more information in the introduction that addressed the 
following: 

• Benefits and uses of evaluation 

• Considerations for measurement in a distributed computing environment 

• Costs of evaluation 



The consensus of the participants was that, while evaluation may be a “good thing,” they all have 
numerous, competing responsibilities and a “good case for doing evaluation” should be made 
upfront in the manual. 



Reporting and dissemination of evaluation results. 
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Overall, the study team obtained much useful information from this site visit We had 
hoped for a more specific assessment of individual measures than we received -- although some 
interviewees agreed to read through the entire draft manual and send us additional comments. The 
emphasis, however, was on issues related to the importance, and use, of ongoing evaluation of 
networked information services and activities. 

An underlying theme throughout the visit was the conflicting views and needs of central 
computing services versus departmental computing services for evaluation and assessment. The 
“distributed” environment significantly complicates the views and needs of various stakeholder 
groups regarding evaluation. Indeed, the responsibilities for computing support and services is 
“confused” at best and typically, faculty and staff are unsure where to go for what kinds of 
services. 

A second underlying theme was the idea that a computing services group, a bigger and 
more sophisticated infrastructure, and more “state of the art” equipment and services are givens for 
an institution such as this one. As one interviewee commented, “Competition among leading edge 
institutions also drives the development of the infrastructure as much as anything else... we have to 
be as good as, or better than, MIT if we are going to compete.” Thus, there is some sense that the 
networking infrastructure will be developed, will be extended, and will be made better -- regardless 
of any evaluation. Such a perspective somewhat limits the need for formal justification and 
accountability of the network. 

A third theme was the identification and description of the various barriers affecting 
successful evaluation of the networking environment which permeated many of the discussions. 

To a certain extent, the academic institution historically has paid little attention to the evaluation of 
traditional activities such as teaching, learning, research, and service. Thus, many of these bamers 
have been in the way for quite some time. Some attention to how best to remove and minimize 
these barriers will be needed if academic institutions are to be “prodded” into conducting such 
evaluations. 
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Finally, this site visit points to the importance of, and need for, a performance measures 
manual that 

• Has flexibility in how it can be used in different institutions 

• Identifies issues that may need to be resolved at individual institutions 

• Is useful for both centralized and distributed computing and networking 
environments 

• Explains the importance of, and need for, ongoing network evaluation in addition to 
a depiction of performance measures 

• Recognizes that different institutions may be collecting data differently, and indeed, 
that within the same institution, they may be collecting the same data differently. 

Clearly, however, the interviewees provided much support for the development of such a manual 
and looked forward to seeing one “in the near future.” 

4.3.2. Public University Site Visit 

A selected public institution was the site of the second visit. Two members of the study 
team visited the site for a two-day jjeriod during which they gave a brief presentation on the 
research project, toured campus computing facilities and libraries, and interviewed individuals 
involved in networking. Twenty-six individuals were interviewed including high-level 
administrators in the computing services group and the libraries, as well as staff members from 
those areas, and members of the faculty and student body. 

General issues related to networking at this institution are presented below. Issues specific 
to the libraries and to the computing services groups follow. These specific issues include: the 
current state of network evaluation and attitudes toward evaluation, and reactions to the proposed 
performance measures and manual. The users’ perspective, from faculty and students, also is 
presented. 
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Interviewees raised a number of issues which they believed were important in 
understanding their approach to networking and evaluation at this institution. 

A client-server approach to network operation and funding . As the university moved from 
a mainframe model to a client-server model of computing, the centralized computing staff 
maintained control over personal computers and the associated network software and applications 
programs. Likewise, the centralized computing staff maintained control over the budget, which 
was growing to support this expanded computing operation. However, this institution’s current 
plan for network deployment and operation, which will take the network into 1999, presents a 
drastic change in philosophy. The centralized computing services group will continue to be the 
server, with the clients being the end users. Both clients and servers will have budget, budget 
justification, and operational responsibilities. Under this plan the total IT budget will continue to 
rise but the computing services group portion will remain constant or decrease slightly. The 
increases in the overall budget will be funded within individual user departments. According to 
one administrator, "This is a bold move in which departments will gain a sense of ownership at the 
expense of creating their own funding mechanisms. This plan is not necessarily the most cost 
effective but will spread the IT expenditures across a variety of more defensible ventures.” 

Shared responsibility is not easily accepted . The people who were interviewed shared a 
number of concerns over the move toward distributed computing and distributed budget 
responsibilities. First, there is concern over the loss of a centralized pool to cover expensive, 
special, and major projects. As one librarian stated, "How will a small library fund the digital 
conversion of its resources to share with others on the network? There must be a centralized pool 
for these major undertakings. " Second, they have concerns over the enforcement of standards. 

The centralized computing group will publish guidelines and standards to insure inter-operability of 
distributed systems, but it is not clear how these guidelines and standards will be enforced and 
what the consequences of ignoring them will be. A final concern has to do with inequalities in 
resources across the campus. While some units may be in a position to fund all the services they 
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need, other units may not One interviewee added, "People don't trust upper management 
planning. They don't believe the central budget will drop. They think CIT will still maintain 
control and shift funding responsibility to the individual departments.” 

New roles for the centralized computing group . Under the networking plan, the centralized 
group has a continuing and major function in academic networking. They will maintain 
responsibility for the campus distribution system, the transmission of information throughout the 
network, and connectivity between the campus network, outside users, and information sources. 
Their responsibility for computer terminal equipment and applications will cease. Although the 
centralized group will offer consultation, advise, maintenance, and repar, on applications and 
terminal equipments, for a fee, they will no longer be responsible for budgeting, funding, or 
obtaining these pieces of the network. 

Libraries as trainers and guides . The library is asserting itself into the role of trainer-guide 
to information content. As stated by the head of library services, "There's a vacuum in this area of 
training, [computing services] provides access but doesn't deal with applications or content. 
Libraries are filling that void in teaching users about electronic information. Some students now 
arrive on campus with experience in the use of electronic sources. It is our job to provide services 
to them while getting others up to speed." 

Budget re-ali pnment. The library systems at this institution are the main users of the 

information network. They have already begun meeting user demand for electronic access by 
realigning their budgets. There is little or no new money available, so less is spent on printed 
materials and more on electronic support services. This realignment of spending will continue as 
libraries move from an attitude of acquiring assets to one of acquiring access. 

Shared resources . There will be a vast amount of sharing of library resources via 
networking in the near future. With new emphasis on understanding and design of electronic inter- 
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library loan systems, the goal of the library is to reduce purchases of shelf materials, minimize 
duplicate copies of holdings, and to focus on electronic access to materials held in other locations. 

Sharing reduces the value of ownership . In addition to funding required to build the 
network, the information content itself has a price. Libraries contain valuable collections that could 
be shared on the network. That sharing however, will reduce the intrinsic value of the asset. 

Some method of compensation must be created. Authors, and their estates, own works that are 
held within the library. These holding libraries must somehow be reimbursed or they will have no 
incentive to share their valued resources. 

Network user fees . There is ongoing discussion concerning the degree to which 
centralized information technologies (e.g. wiring, routers, and Internet access) should be budgeted 
from the common construction fund, much like roadways. In a move designed to alleviate a 
portion of the current IT budget shortfall, students at this institution will be assessed an IT user fee 
of $60 per semester. This new fee will not initially provide any new services. The monies 
collected will be used to expand the campus data network and to offset some existing departmental 
budget constraints. The allocation of these monies between CIT and individual departments has 
yet to be decided. The administration expected student protests to begin as soon as students 
arrived on campus. This will, in turn, generate requirements for analysis and defense of the 
proposal and for the services provided; a further justification for network measurement and 
evaluation techniques. 

Informal evaluation . There is no formal, ongoing program of evaluation of network 
technologies, resources, and services offered through the campus libraries. However, library 
administrators and staff are interested in measuring the degree to which they are meeting their 
users’ needs. As one participant said, “If you’re constantly asking yourself questions then there’s 
a better chance that you’re doing something right than if you basically say, ‘Hey, we’re doing 
great. ’” Currently, library staff conduct informal evaluations in a variety of ways and rely on 
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users, faculty, and librarians at other institutions to provide information which can be used in 
evaluating the network. 

Observations . One participant described the library’s primary approach to evaluation as 
“keeping our eyes and ears open.” Heavy use of network terminals, defined as those times “when 
we get people piling up to use them,” is interpreted as a positive evaluation of the network. From 
such observations, staff infer the users’ satisfaction with the resources and the need for additional 
terminals. Repeated problems with certain network services, or the lack of use of certain services, 
cire interpreted as negative evaluations. 

Staff members added that there have been instances when their perceptions were found to 
be in error after observations were made. “We thought that the big use for online reference would 
be evenings and weekends. [We found] weekend use was virtually nothing, there was some 
evening use, but most of our questions came in during the hours that we were sitting at reference 
desks. Sometimes conventional wisdom and what actually happens are at odds.” 

Use statistics . To a limited extent, statistics on use of network services and resources, 
specifically those resources available via the OPAC, are collected. “We generate monthly statistics 
on the types of searches that eire being done and the databases being searched.” Those statistics 
have been used in making decisions about which resources to offer and in what form, but they 

have not been analyzed extensively. “We’ve used [the statistics] to a certain extent to manipulate 

the configuration of databases and to maintain certain index and abstract products.... But [the 
analysis] is on a transaction basis. It could be one person doing a million searches for all we 
know....I’m not satisfied that we’ve done nearly enough evaluation.” 

Focus groups. Library staff also have used focus groups as a means of evaluating the 
electronic resources they provide. After a series of focus groups of students, conducted to 
determine the appropriateness of online databases offered, a number of changes were made in both 
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the content and means of access to those databases. The following excerpt from a report on these 
focus groups identifies several areas where changes were needed: 

“...more graduate researchers would be inclined to use the system if a greater span of years 
was offered.. ..Better publicity, instruction, and perhaps even signage seems to be 
indicated.. ..Release 5.0.2 will facilitate progress in this area. Help screens will be better 
and an improvement in index screens will be apparent. A less cluttered display may 
promote more reading of instructions and, therefore, more self reliance.” 

Faculty evaluations . Staff also rely on faculty evaluations of student work as an indirect 
evaluation of the network services they provide. One librarian described a recent situation, brought 
to her attention by faculty, which indicated that students were in need of additional training in the 
use of online resources. “The quality of the students’ papers was going down. They weren t using 
secondary sources. They were only using sources they found online.” As a result, changes were 
made in the academic program and in library instruction to educate students in the differences 
among information resources. 

Peer institutions . Finally, staff also rely on librarians at other institutions to provide 
feedback and recommendations concerning network resources and services they offer at their 
institutions. In making a decision to offer one of a number of similar resources, library staff might 
call peers at other university libraries to ask for their opinions of the resources. 

Overview of barriers to ongoing networking evaluation . There are a number of reasons 
why there is not a formal, ongoing program of evaluation at the libranes. They include: existing 
attitudes toward technology and toward evaluation, a lack of resources, and organizational and 
environmental conditions. These barriers are elaborated on below. 

• Positive attitudes toward technology . There is an underlying assumption that 

networking is a good and necessary thing. This assumption, shared by those at the 
highest level in the organization, is driving development of the network and is 
interpreted as a signal that evaluation in order to justify the network is unnecessary. 
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As one staff member explained, evaluation for the purpose of obtaining technology 
resources has not been necessary: 

“We essentially lost acquisitions money this year with one hand while they 
were giving us automation money from the other hand. They’re saying basically 
that technology is the way we have to go.. ..There’s the general perception upstairs 
that technology is good and that we should move toward it. It’s a means to access 
information that we’re not going to be able to buy or locally own.” 

Staff members indicated, however, that this perception could change. Until 
now, technology has been acquired and implemented using “an ad hoc budget” 
That is, there have been no regular budget allocations for technology in the libraries. 
They speculated that, once the cost of technology is built into the library budget, 
there will be institutional expectations for evaluation and justification. 

• Results of previous evaluations have not been useful. Staff members said they have 
not found the results of previous evaluations to be useful or applicable. 

“I don’t think [past studies] have told us much we didn’t know sort of 
instinctively.” Also, “It’s hard to do anything with what you get Y ou get mixed 
signals, there isn’t one clear message. Everyone’s so different. They use things 
differently, they learn a different way. [This] makes it hard to do anything with 
what you hear. People approach [network technologies, resources and services] 
different ways. Their sophistication is different... If you really studied something 
you would see such a range of needs and whatever, you would be overwhelmed. 
There wouldn’t be a [single] solution.” 

• Lack of personnel. The library’s online public access catalog (OPAC) collects data 
on use of the system. However, the resources available to analyze the data are 
limited. “We find that we generate these log files but that the manipulation of those 
files could easily be a full-time job for a person. Therefore we basically do ad hoc 
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requests.... The log of public searches is a way to evaluate the system somewhat. 
The problem is we can’t do that every day and it’s very dangerous to draw a lot of 
conclusions looking at a day’s worth [of data].” Also, there is no MIS to handle 
the data collected and no overall evaluation strategy within which to analyze the 
data. 

• Lack of research expertise . Furthermore, it’s not just a matter of having additional 
staff to analyze data, but rather it’s a matter of having staff with the appropriate 
skills and experience. 

“What a lot of us are struggling with here is the fact that research is not easy 
to do. Any research requires understanding of the methodologies involved, it 
involves careful preparation and thought to design it well and work to analyze and 
interpret the data.. ..we’re all practicing professionals with day-to-day 
responsibilities for managing services and resources and not all of us have a formal 
background in research methodology. And there’s a learning curve to do it well.” 

• Lack of precedents for evaluation . Like other organizations studied for this project, 
there is not a history of evaluation at this institution. It appears that many of the 
barriers to evaluation of network resources and services also were barriers to the 
evaluation of other resources and services. As one participant said, “It’s a little 
strange to ask what do we do to evaluate things now when this is a question that 
was never asked in a pre-network environment.” 

• Organizational structure . Other barriers to network evaluation may be products of 
the structure of the organization. One problem is that the structure is not always 
clearly defined. Deciding where one element of the structure stops and another 
begins is not always an easy matter. For example, it is not clear, in the case of a 
statewide project, where the responsibilities of this institution end and the 
responsibilities of other state institutions begin. One participant explained why this 
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is the case. “We’ve been trying to blur the institution boundaries as a whole. 

We’ve been talking about statewide purchasing of databases. We’ve learned that 
we can do 50 % of our interlibrary loan within [the system of state institutions].” 
Another problem is the complexity of the organizational structure, which is 
mirrored in the design of systems to provide information, and which complicates 
data collection for evaluation. “Any system that we have is going to be complex if 
we replicate the organizational structure of the library and that’s what we were 
trying to do.. ..That stands between us and getting easy information.” 

• Rapidly changing environment . Another barrier to network evaluation is the speed 

at which new technologies become available and old technologies become obsolete. 
Staff members find it difficult to anticipate new technologies. As one person 
described the problem, “We don’t know what [research and development] is going 
on in the rest of the world. I don’t know what [research and development] is out 
there and is going to float on the scene any minute.” And this rapid development 
and obsolescence of technologies makes evaluation difficult. “When you’re talking 
about networking, things happen so fast that it’s hard to put that kind of effort into 
evaluation of something that, by the time you get the evaluation done, you may be 
on to something else or something has changed.. ..We can’t spend three weeks 
developing a report that’s useful but it’s useful in regard to a dead technology.” For 
example, “I don’t want to spend a lot of time evaluating CD ROMs because I ’m 
already looking at networked databases.” 

The library staff members who were interviewed agreed that: there are a variety of problems 
with evaluation of the network, previous attempts to evaluate have not been terribly successful, and 
the situation at this institution is typical of others. As one person described the situation: 
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“My feeling about evaluation is that it takes a lot of time, you’re never sure you’re doing it 
right, and then I’m always finding myself in situations where after you’ve done it you 
never actually do anything with it or let it really inform something you’re doing. I think 
there are barriers somehow... psychological barriers... organizational barriers....! don’t 
think it’s imique to this situation.” 

To a certain extent, the barriers to evaluation which the library staff members identified reflect a lack 
of understanding of the multiple purposes of evaluation. The staff members viewed evaluation as a 
means to justify resources and services offered to patrons and as an aid in decision making about 
the number and variety of resources and services to offer. This limited view of evaluation also is a 
function of the administrative level of those interviewed. Most of the people interviewed were 
involved directly in patron service and support. However, at a higher administrative level, there 
was a better understanding of evaluation and an appreciation for the need to evaluate networked 
resources and services in a larger organizational context. For example, the library director 
commented that she has been asked questions concerning cost benefit analysis of certain resources 
and services; questions for which she does not have, and would like to have, the information to 
answer. 

Study team members provided participants with copies of the draft performance measures 
manual in advance of the site visit and interviewed these participants individually and as a group to 
get their feedback on the manual. Their comments, about the manual in general and about 
individual measures specifically, are summarized below. 

Usefulness and usability . Participants agreed that the proposed measures would be useful 
and that the instructions given in the manual for collecting and interpreting the data would be helpful 
in using the manual. As one person stated, “Personally, in my day-to-day work, I could use a lot 
of these things and I like the way you define things and operationalized things. I could use some of 
these answers.” 
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Someone else commented that the instructions and advice given in the manual concerning 
sample selection would be particularly helpful. “I did notice that you addressed sampling and that’s 
appropriate. There’s no way you could measure everything. I think sampling is a good thing to 
include.” 

Another participant observed that the proposed measures are especially useful when 
compared to other measures that are currently available, such as statistics published by ARL. 

“There were a lot of sets of statistics that were published aimually that were good for a 
while, but now they’re not real helpful anymore because they’re not measuring these 
things. They’re not asking the modem questions, [concerning the impacts of information 
technology and networks] unfortunately. They just count, how many of these do you 
have, how many of those. They’re quite detailed but they’re not useful.” 

Inclusiveness. Participants also responded positively to the inclusion in the manual of 
performance measures for groups outside of the libraries, such as computing services. “What I like 
in this document.. .personally, as I try to promote things on campus. I’m very dependent on the 
performance of [computing services]. When I tell someone to call the help desk, I say to myself, 
‘Are they really going to get help? Will they find a courteous person? What’s the response time 
going to be there?’ And there’s a lot of that built in here.” 

However, some participants said that the measures failed to identify all groups involved in 
networking, including other on-campus groups, such as schools and colleges which work with the 
library to provide network resources and services. 

“It still seems like an approach where you’re counting things for some discrete unit, some 
institution. But if I have access to someone else’s institutional resources, that doesn’t come 
in here very easily. If I was trying to apply this to the library it doesn’t count that we are 
trying to do things with the engineering department and we have access to their resources in 
a way that maybe is not even specified.” 
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Others suggested that the measures might be amended to reflect the participation of vendors in 
campus networking. For example, if a network service offers access to a variety of databases, 
provided by a variety of vendors, there should be some measure which distinguishes between the 
databases and permits evaluation of them separately. 

Basic measures . Participants agreed that the basic measures proposed, the Count of 
Network Users and Annual Information Technology Expenditures, are of primary importance and 
that the definitional problems associated with these basic measures can only be resolved by 
individual institutions. 

Need for measures of connectivity . Participants suggested that other useful measures 
would be ones which would indicate the degree to which faculty and students have access to the 
network and where they have access to the network. They suggested that what’s needed are 
measures “...that would give institutions a way of saying how we are doing in general in getting 
our community into this electronic network of information resources. The extent to which the 
community is connected to the network... how many people have some sort of connection from 
their offices or dorm rooms or whatever. We’re still struggling with that on this campus. It’s a big 
issue— how quickly can we get all the offices wired. So some basic measure of connectivity of 
faculty, staff and students to the network.” 

Need for gross figures. Participants suggested that general statistics on use would be more 
powerful in their institution than statistics on who is using what and when they are using it. 

“Figures like the hundreds of thousands of transactions that we see everyday against the system are 
figures that we use and figures that are appreciated because of their magnitude. But 1 don’t think 
[decision makers] stop and think, well who is doing this? Is it the faculty? Is it the students?” 

Need for comparative institutional data and network configurations . Participants expressed 
concern that it would be difficult to compare various institutions using the proposed measures. 

‘The whole tone in education now is one that sees technology as something that is good and that 
makes this school competitive with other schools. A lot of eveiluation is how we compare.... One of 
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the first things we were asked when we started thinking about moving to Unix was ‘Well who else 
is think ing about that?’ And when you say Columbia and Cornell are thinking about that.. .that 
means a lot.” The kinds of information the library staff would like to have about other institutions 
include, “Information about an internal network, a server-based system, who’s on it, what does it 
have on it, can you dial into it, what else can it get to, things of that sort, what are they costing...” 

Need for data on use of specific resources . Library staff also are interested in measures of 
the cost and use of specific information resources. “If I had data on the [use of] journals in my 
collection and I had data on the [use of] databases that we had mounted on the network and I could 
compare the use and the cost per use that would be something that I could use very, very easily. 
Now, when I talk to faculty they say this is great, we’ve got to have it. But when I say it’s going 
to cost $20,(XX) and we’re going to have to cut nine journals to do it, they say oh.. .but if I could 
show much more specifically that we were getting much more bang for the buck it would be worth 
it.” 

Additional measures . Participants stressed that the proposed measures tell only part of the 
story. In addition to the measures in the manual and the measures suggested above, they would 
like to see the manual include the following: a user survey, methods for collecting and analyzing 
qualitative data, methods for collecting and analyzing anecdotal data, and methods for collecting and 
analyzing longitudinal data. (Note: The first three items listed above were in development at the 
time of the site visit but were not available for review at the time these individuals were 
interviewed.) 

Need for measures of quality. A final concern had to do with measurement of the quality of 
network services and resources. “How do we make the jump from quantitative numbers to 
numbers that can help us evaluate the quality of what we’re doing? It’s easy to say we’re spending 
this many dollars to provide this many workstations to provide access to this many databases, but 
so what?” However, the individuals interviewed were unable to describe specific measures of 
quality which they would like to have. 
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Likelihood of using the manual . In spite of participants’ positive comments about the 
manual, there was some doubt that they would actually use the manual, even if the changes they 
proposed could be incorporated. “In an ideal world, theoretically, yes, I think we should and need 
to be collecting these kinds of data and analyzing them on a regular basis. But, realistically, I don’t 
think... I think we’d have a hard time convincing ourselves and those above us that the work and 
effort involved in collecting these in a systematic and effective way would have the highest 
priority.” 

The major concern of participants was the resources required to actually implement the 
proposed measures. “A lot of this would be wonderful, to have the information. I’m just 
wondering how much we’d have to invest. My programmer could work a whole year on this, 
setting it up.. ..But realistically, unless we had a research officer that was hired to be on this campus 
to do this kind of thing, I don’t think those of us who are here would do this.” 

Preference for broad guidelines instead of specific measures . Participants suggested that 
another type of approach would have a greater likelihood of being implemented in their institution. 
One person described the preferred evaluation instrument as, “A set of guidelines. A series of 
questions [to be used as] a sort of accreditation process. A set of self evaluation questions you ask 
yourself periodically. And if you can answer yes to most of those questions, then you’re probably 
doing a good job. Rather than counting things you’re asking yourself a series of questions. ‘Are 
we doing these kinds of things.’” Participants pointed to the CAUSE/EDUCOM Evaluation 
Guidelines for Institutional Information Technology Resources as an extunple of this type of 
approach. 

Preference for small-scale projects. Also, participants suggested that a narrower scope 
would be more manageable. “I think [what we need are] periodic, smaller-scale, focused research 
projecte, with careful methodology, from time to time, that one can then publish or get in the 
literature and cite as a backup, that it is true that these technologies are having an impact. Those 
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would be focused, smaller-scale projects, rather than trying to, on a very large scale, collect all this 
kind of data and try to evaluate it” 

While the individuals who were interviewed at this site were generally supportive of the 
team’s approach to evaluation, the manual, and many of the individual measures, they also 
expressed some reservations. Their primary concerns were that: the proposed measures don’t 
capture contributions by all parties involved in providing network resources and services, they may 
not have the resources to implement the performance measures described in the manual, and they 
don’t have a clear understanding of how to interpret and apply the results. In order to address these 
concerns, the study team would need to: 



Expand the measures to include all networking participants, such as vendors and 
various colleges and academic departments on campus, or provide guidelines 
demonstrating how this might be done. 

Suggest, in the introduction to the manual, how institutions may use some of the 
measures to compare their performance to other institutions. 

Illustrate in the manual how these measures might be carried out on a longitudinal 
basis. 

Explain how the results may be interpreted and applied. 

Empheisize that the usefulness of any given measure will vary from institution to 
institution. Users of the manual should be encouraged to select measures from the 
manual which they feel would be useful and should be reminded of the value of 
individual measures. 



The academic network at this institution is on the brink of transition from a centralized, 
tightly controlled environment to one with a distributed architecture, distributed funding, and 
multiple interdepartmental operational processes. In the past, the centralized computing group has 
been responsible for acquiring, providing, and evaluating all information technology resources at 
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the university. The change to a highly distributed environment will enhance the acquisition and 
provisioning of network components but will complicate the evaluation process. 

Budgeting and funding . As documented in the organization’s technology plan, and 
discussed during the site visit, central computing historically has been the major provider of 
financial support for information technologies for the entire campus. The current annual IT budget 
is at $20 million with 73% coming from the central unit. The 1999 projection of $30.4 million will 
reduce the central contribution to 46%. As CIT's budget responsibility drops below 50%, the 
ability to maintain control and to evaluate the total cost of the academic network will become more 
difficult. Interdepartmental teams will be formed to provide the required evaluation. 

Network traffic statistics . The primary functions of the central group include acquisition, 
provision, and operation of the network. Evaluation techniques have been developed to facilitate 
these functions. Traffic measurements throughout the network that contains more than 100 sub- 
networks include daily counts of bytes, frames, packets, and modem port connections. These 
measures are primarily used for daily network operations and trouble analysis. They are also used 
to assess trends in sub-network activity that may dictate the need for network reconfiguration, and 
to ascertain needs for equipment additions. Network traffic statistics are posted daily on the 
Internet and may be accessed via the Worldwide Web. This posting provides an excellent storage 
environment, as well as easy access by personnel doing testing, maintenance, and network 
pl anni ng This Web Server arrangement has been written up and received high praise in trade 
publications. 

It would be advantageous to central computing to know the originating location of modem 
traffic to the campus. Since these calls originate within and travel over the public telephone 
network, the Automatic Number Identification (ANI) feature, the same feature used for incoming 
caller ID, could be used. However, studies have found more than 60% of the users have disabled 
this feature on their telephones negating this data collection opportunity. 
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Network extensiveness . The study team was surprised to find that only 50% of the staff 
and faculty offices at this institution are wired for direct network access, and that there are no plans 
for direct access from any of the 8000 dormitory rooms. Central computing does have a specific 
plan and timetable for the wiring of the remaining offices. The modem pool, which provides for 
access from the "un-wired" offices, dormitory rooms, and off-campus users, is under constant 
evaluation and was recently increased in size. The high speed, fiber optic, backbone network 
connects sub-networks, an off-campus hospital network, and the Internet, through distributed 
routers. 

Quality assessment . According to the central computing staff, "The foremost quality 
measure is a network functioning without blockages and delays." From a maintenance and 
operations viewpoint, this is a viable measure. From the users’ perspective, quality may be a 
totally different measure. Central computing has conducted online user surveys, but offered no 
specifics concerning content or results. Peer evaluations have supported the creditability of the 
network operation. Other universities and colleges have relied on the expertise of the central group, 
in network planning, design and technical evaluation. 

In s ummar y^ a sophisticated system is in place to measure the day-to-day operation of the 
network and to pinpoint locations of network troubles. Since only 50% of the administrative and 
faculty offices on the campus are currently networked, there is also an ongoing measurement of 
progress in network extension. Funding for network deployment is monitored in both the 
centralized operation and within departments. Beyond these measures, there is no ongoing 
evaluation of network effectiveness or impacts on the mission of the institution. 

Overview of barriers to ongoing networking evaluation . As shown above, evaluation is 
done in those areas necessary to keep the network fimctioning on a daily basis and within the 
budgeting and fimding process. Beyond that, there is a lack of a formal, on-going evaluation 
scheme to assess impacts on the academic environment, for the following reasons: 
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• Assiimntion that "networks are good . "Until very recently there was little concern 
about the money being allocated and spent on computing and network systems. 
The belief that education and technology inherentiy compliment each other fosters 
an attitude of minimal concern for evaluation techniques. 

• Assumption that “intuition is sufficient and probably better than numbers." Central 
computing personnel feel competent in network planning and design and use 
quantitative data only when necessary to analyze and fix problems. Basic trend 
information is used to show growth in support of the idea that the academic 
community needs and uses the network. Beyond that, equipment selection and 
deployment is done by intuitive processes. "The real question in measurement is, 
'What gives you the biggest bang for the buck?' The idea of 'biggest bang' is 
intuitive and not created through evaluation." 

• Lack of resources . Time, money, and personnel are not available to do evaluation 
beyond that necessary to build and operate the network. "Evaluation of budget 
requirements and network bottienecks take most of our resources." "There are 
always three specific problems associated with measurement: it takes too much 
time, you are never sure if you measured the right thing, and finally, the 
measurements are not used for anything." 

• Lack of knowledge on how to evaluate . "The research questions proposed within 
the study are excellent and important but can we really find any way of answering 
them?" This statement portrays the current state of affairs in network evaluation. 
"The people assigned to do evaluation are usually not trained as researchers. 
Developing evaluation schemes and interpreting the results requires understanding 
of methodology, thoughtful design, and a large amount of effort. Knowledge 
pooling and sharing of successful evaluation activities on other campuses will 
reduce this barrier." 





113 



Evaluating a moving target . "The speed of change is awesome." The environment 



is changing so fast that it is difficult to develop a measurement. "By the time you 
have plarmed, implemented, and analyzed data in a measurement area, the need for 
the analysis will have passed. Today there is no such thing as long-range planning. 
Unexpected applications can greatly distort any measurement activity. We would 
be better off spending time trying to understand what research and development is 
going on rather than trying to measure what we know we have." 

• Reduction of central control . As end users gain more control over the budgeting, 

fimding, and equipment selection processes, the ability to evaluate the overall 
network will decrease. The boundaries between departments and areas of 
responsibility are no longer clear. "We will need definitions of what needs to be 
evaluated, why, and for what purpose. It will be hard to generalize from the results 
created by people of all different levels of technical sophistication, and different 
ways of looking at the same results. Without some department or individual calling 
the shots, no useful evaluations will result" 

• Inequalities in resources and power. As the distribution of funds for network development 
and use becomes more widely dispersed, an environment of "power pxDckets" will emerge. 
Some departments will be more able to allocate existing resources to technology and to 
obtain outside funding through business partnerships. This will cause an evaluation 
dilemma in which disclosure of success in network activities may result in decreased 
funding from the common p)ool. 

• User accounts . The current procedure at this institution is to provide each incoming student 
with an E-Mail user account during the registration process. There is no ongoing process to 
purge accoimts of graduates or "dropouts" and therefore a count of E-Mail accounts would 
be an inflated number. UNIX E-Mail servers are capable of coimting individual account 
activity but such data is not normally accumulated as no need has been shown. 
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In summary, there is a general feeling that networks are good and therefore their impacts 
must also be good. This feeling presents a barrier to additional funding and manpower without 
which measurement, beyond that required for daily network operations, will not be implemented. 
The speed of change in technology, user behavior, and allocated network responsibilities make 
measurement difficult. The unequal distribution of resources, both physical and knowledge-based, 
has created network power pockets. There is lack of desire on the part of individuals within these 
power pockets to evaluate, disclose, or redistribute their resources. The best sources of evaluation 
will come from the user community in its assessments of impact, especially in the areas of network 
accessibility and quality. 

Members of the computing services group had the following reactions to the draft manual 
and the measures within it: 

Basic measures are important . The consensus of the interviewees was that basic measures 
are important to set the foundation for specific measures leading to the assessment of impacts. 
Measurement of network extensiveness, growth, number of users, and expenses associated with 
network deployment and operation were deemed to be the best starting points. 

Measures require associated benefits. "We cannot use the performance measures manual 
unless we are given extra resources. There is just no time to do this stuff. We would have to know 
how much it would cost and what the benefits would be for following the manual." This statement 
highlighted the need for the study team to not only detail the measurement procedures but also to 
articulate potential benefits that would result from such measures. 

Minimal need for fine tuning . There was general agreement that the proposed measures of 
cost, number of users, network traffic, and support functions all provide a starting point for 
network impact evaluation. There was concern that fine tuning of these measures, (e.g. counting 
each and every user, or each and every packet) could complicate the process to the point of 
destruction. "It would be better to establish a consistent measurable base from which to do trend 
analysis than to make sure that we counted every user and every login." 
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Traffic measures in the manual are “right on target .” The computing group praised the 
traffic measures section of the manual. They currently take counts of bytes and packets in much the 
same manner as the study team proposed. "We agree with the proposal for sampling data. Constant 
data recording provides just too much information. At [this institution] we do the sample counts at 
ten minute intervals." 

The traffic measures at this institution are used to facilitate daily operations and for simple 
growth analysis and equipment planning. They are not specifically used to assess network impacts. 
They do not count traffic to specific applications. "Counts of total network packets, and counts of 
log-ins to UNIX servers are quite easy to obtain. Beyond that, to assess the application associated 
with individual packets is expensive and complex. " They acknowledge that such data could be 
gathered by providing "network probes" or "sniffers" at specific network locations but the costs 
overshadow any perceived benefits. "Such devices cost in the range of $10,000 each, and we have 
100 nodes in our network. We use these devices for trouble analysis but to deploy them for routine 
data collection would be cost prohibitive. " 

Since this institution made a decision not to wire dormitories for direct coimection to the 
campus network, the modem pool takes on added significance. Similar to what the study team 
found during the other site visit, people at this site said, "No matter how much we expand the 
modem pool, we reach saturation levels. Many students will try coimecting from their rooms 
before using the on-campus public clusters. " It is considered important to monitor modem pool 
activity to look for sources of trouble but there is no policy to keep modem traffic below saturation 
levels. 

Likelihood of using the manual . The basic traffic measures to assess network day-to-day 
operations, and to measure modem utilization, are probably in use in all but the smallest network 
environments. The use of these measures for trend analysis is fairly simple and is used to support 
funding and network expansion plans. Beyond that, the interviewees felt that "Additional gathering 
of quantitative data won't tell you much. What we need next are survey tools." 
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Need for qualitative measures . The interviewees expressed the feeling that cost measures 
and traffic measures are necessary for the daily operation of the network, but it will be subjective 
measures that indicate the real impacts on teaching, learning, and research. “It is by monitoring 
user demands for services, watching waiting lines at public clusters, and administering surveys that 
we will reveal the impacts. The qualitative, anecdotal data may not be as fundamentally sound as 
quantitative data but it will carry more ‘weight’ in the evaluation of impacts." 

The quantitative traffic measures outlined within the manual were exact parallels to those 
currently is use at this institution. They are necessary for the daily operation of the network. 

Beyond that, the proposed measures will require additional effort and increases in required funding 
and manpower. It is unlikely that these measures will be used unless significant benefits can be 
articulated. At this time, computing personnel feel that they have a sufficient handle on the budget 
and that provision of a user ID for every incoming still is an adequate measure of the user 
population. What they desire in terms of further network assessment are inexpensive, qualitative 
tools to get the views of the user community. 

In addition to the interviews with library and computing staff, the study team conducted 
group discussions with members of the faculty and the student body at this institution. Although 
they are less involved, generally, in evaluation of the networking technologies and services, their 
perspective is important to include. Following is a summary of issues raised by these groups. 

.Shared resources increase the amount of available information . The networked environment 
provides access to more information than otherwise possible, including original manuscripts, 
musical compositions, and medical records. The information on the network is better organized 
and is changing the very nature of literary research. The work of the student or researcher is no 
longer constrained by materials in their possession or in the physically accessible library. With the 
network, world-wide searches can be completed in minutes. The key to all these benefits is 
sharing. 
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Not all activities are easier "on-line. " Sharing of resources provides value in many but not 
all phases of the education process. One student who gave high praise to network benefits said, "It 
makes my research work much easier.” On the other hand, the reading of lengthy texts online is 
not desirable. He added, “I don't believe anyone would want to read Moby Dick on-line." 

Enhanced classroom presentations . Within the classroom, the traditional "chalk and talk" 
lecture is being replaced with methods based on technological enhancements. This institution has 
an ongoing modernization program to equip all classroom facilities with electronic podiums. These 
consoles provide instructors with access to overheads, slides, and videos, as well as connectivity to 
the network. All presentation materials are digitized and projected with high quality. The network 
connection allows real-time information searches, downloading of soimd bites and video clips, and 
distance learning opportunities. 

Enhanced communications activities . From the students’ perspective, networking has 
provided a convenience that they believe would be difficult to do without. In addition to improving 
the content of the lecture, networking facilitates communication between students and their 
instructors. Students said that faculty are more responsive to email than they are to telephone calls 
or office visits and that email responses are of higher quality because time is available within the 
communication process for research of the question and formulation of the response. Students also 
use the network to communicate with each other, with family, with friends, and with professional 
organizations and businesses when engaged in the job search process. The network also provides a 
valuable link to the institution after graduation. 

Physicians and medical students were initially attracted to the network because of its 
novelty. They became extensive users after discovering that it provided fast access to current 
clinical data and enhanced communication with colleagues in remote locations. They tend to use e- 
mail for distant communications but prefer the telephone as the primary medium for local work. 

This second site visit, to a public institution, provides an interesting contrast to the first, 
which was done at a private institution. These institutions are alike in that they face many of the 
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same barriers to evaluation. However, this public institution, and perhaps other public institutions, 
face a different set of administrative issues and budget constraints than private institutions face. 

The budget constraints faced by this public institution appear to create a greater need for, and 
impetus to develop, a network evaluation program. This second site provides an excellent example 
of how network traffic can be monitored on an ongoing basis and how measurements of network 
traffic can be aggregated and used to support decision making. 

4.4. Final Focus Groups 

After the site visits, the study team revised the assessment manual and the measures, based 
on feedback from the visits. Two final focus groups were conducted to “test” the revised manual 
and measures. The first was held at the same university as the initial focus group of the project and 
the second was held in conjunction with the 1995 Fall Meeting of CNI, in Portland, OR. 

4.4.1. University Computing 

Individuals from the university’s computing services group were asked to read and 
comment on the revised manual and measures and then discuss their reactions at a focus group. 
Seven individuals participated in the focus group. Following is a summary of their comments and 
reactions: 

• Explain assumptions . Participants suggested that the team explain in more detail the 
assumptions about networked environments that the manual is built on (e.g., the 
degree of centralization assumed). 

• Centralized vs. distributed approaches . Participants suggested that the team 
distinguish between centralized and distributed approaches to networking and 
outline the implications for assessment. 

• Value of Surveys . Participants agreed that the user survey included in the manual 
was an important addition which is likely to be widely used. 
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• Benchmarking . Participants suggested that benchmarking would be a valuable tool 
to add. 

• F)i stance, education . Participants suggested that there will be a growing interest in 
distance education and the relevant measures might be expanded. 

Additionally, the participants commented on measures which would be particularly useful in their 
networked environment, problems they might encounter implementing the measures, and possible 
solutions to those problems. The information collected during this focus group was used to once 
more refine the manual in preparation for the final focus group. 

4.4.2. CNI Portland 

The purpose of this focus group was to obtain feedback, from high-level professionals in 
academic networking, knowledgeable about assessing academic networking, and with “hands on 
networking experience” on the revised draft of the performance measures manual. 

Criteria for selection of focus group participants were as follows: individuals 

• plaimed to attend the CNI Fall Meeting, 

• were interested in the project, 

• agreed to review a copy of the manual prior to the focus group, and 

• were employed at a high level of either academic library administration or academic 
computing administration. 

In an instructional letter to the participants, the investigators stressed the importance of developing 
specific and detailed recommendations that could be used to improve the manual. 

Initially, each participant was asked to introduce herself or himself and to comment 
generally on the manual. Following that, the investigators used the broad questions listed below to 
guide the discussion. The investigators also encouraged the participants to raise any additional 
issues, problems, or questions not addressed by these broad questions but which they believed 
were relevant. 
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• To what degree are the quantitative measures, in part II of the manual, appropriate, 
feasible, and useful in academic institutions? 

• Do you see problems associated with the implementation of the quantitative 
measures? If so, can you propose solutions to these problems? 

• To what degree is the manual appropriate for use across a campus? In individual 
departments of an academic institution? 

• Are the explanations of the qualitative research methodologies which may be used 
in assessing performance of academic networking adequate? 

• Are the topics listed for investigation using the qualitative methodologies 
appropriate? 

Two members of the study teeim moderated the discussion and took notes during the session. The 
same team members did a content analysis of the notes to identify problems and issues to be 
resolved in the next draft of the manual as well as other areas needing improvement. The 
discussion at the session provided a rich range of information that would assist the study team in 
revising and improving the manual. These suggestions covered a number of topical areas as out 

General introductory comments by the participants at the beginning of the focus group 
characterized the project as timely and relevant to their understanding of the current situation in 
academic networking. Participants offered the following evidence to support the need for 
performance measures in academic networking: 

• One participant’s university currently is developing 21 critical measures against 
which performance will be measured and which will be tied to funding. This 
university’s library will be responsible for developing performance measures for 
their unit. 

• One participant is currently involved in the development of a symposium on 
academic networking and performance measures. 
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• One participant reported having used quantitative measures to evaluate networking 
and plans to apply qualitative measures in the areas of user service and satisfaction. 

• Several participants reported that their institutions are interested in evaluating 
networking performance and are prepared to do so, provided they can find the 
appropriate tools. 

• Several participants expressed a desire to use the performance measures manual, or 
parts of it, immediately. 

Participants recognized that developing such a manual would indeed be difficult. Further, a number 
of issues and problems would likely not be resolved by the manual. Nonetheless, the project 
would “move us forward” in this area and should be completed as soon as possible. 

Quantitative measures. The next topic discussed was the number and type of quantitative 
measures that had been outlined in the manual. In this section of the discussion participants 
suggested that language be incorporated for measures using logs that data from the logs should be 
compared to “sniffers” as a validity check. The use of sniffers for a range of data collection 
activities was met with different responses from different participants. Some thought they were 
excellent tools, others were not as certain. Participants made the following suggestions to improve 
specific measures in the manual: 

Measure: Count of Network Users 

The use of email addresses to count network users will be problematic in institutions that have 
multiple distributed email systems or systems which support a range of network use without email 
access. A count of the number of Internet connections to PCs on campus, if there is universal 
network access at the institution, could more accurately reflect the number of users. The various 
ways of coimting users should be listed with the advantages and disadvantages of each. 
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Measure: Annual Information Technology Expenditures 

Describe cost categories that are in greater detail and that are similar to cost categories in other 
widely used data collection instruments (e.g., NACUBO, IPEDS, and CAUSE). Users should be 
reminded that cost structures will differ from institution to institution, costs are often distributed, 
and there will be ambiguities in cost categories. Provide users with greater guidance in finding out 
how to identify the various costs for IT on the campus. Expand this measure to include service 
costs, e.g., the cost to provide a network connection [note that such measures are included 
elsewhere in the manual]. Distinguish between costs for baseline or core services and billable or 
add-on services. The sense of the group was that the total IT costs should not include the billable 
costs to departments or other units on campus. These measures should differentiate between 
services which users receive free of charge and services for which they are billed. 

Measure : Online Library Catalog Measures 

In some institutions, it may not be possible to separate online library catalog use from other uses of 
a workstation. There are different types of online library catalogs connected to different types of 
networking configurations. More detail on types of issues that will have to be addressed should be 
presented. 

Measure: Help Desk 

This measure, as currently written, assumes that the help desk is centralized. The definition should 
be expanded to include distributed help desks and instructions for evaluating these distributed help 
desks should be included. A discussion took place about having separate measures for “central” 
versus “distributed” help desk services. Another concern raised here was the inappropriateness of 
assessing the help desk on topics or problems that may not be the responsibility of the local help 
desk. Increasingly it is difficult to determine where the boundary occurs between the campus 
network and the external network in terms of “help.” 
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Measure: Network Training 

Training measures are “smooshy” (i.e., they may not be valid). Evidence of the adequacy and 
efficacy of training may show up in other measures. Therefore these measures may be condensed 
or eliminated. Also to be considered are the range of individuals and units that provide some type 
of training, e.g., the library. 

In addition to these suggestions to improve measures, participants proposed the following 
new measures: 

Measure: Count of Non-Users 

Definition: members of the authorized service community who do not use the network. 

Measure: Count of External Use 

Definition: Many institutions will be interested in identifying external groups which they support 
(e.g., public schools and professional associations). This may be especially important in rural 
areas where other means of network access are not available. It may be more feasible to measure 
external use of the network than external users. 

Measure: Cost to Support External Communities 

Definition: amount of local resources dedicated to provide users outside of the institution with 
access to unique information or services. For example, such a resource would be a web page 
which is supported by the university and accessed by individuals around the world. The rationale 
for this suggestion is that heavy access to such resources will affect performance of the network 
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and that members of the university community will be subsidizing use by others outside of the 
community. 

Measure: IT Productivity 

Definition: The degree to which outputs of the institutions have been affected by networking 
[unfortunately, specifics on what these outputs are were not detailed]. 

Measure: Savings through Resource Sharing 

Definition: A financial measure of savings achieved by sharing hardware, software or staff with 
other institutions, and by participating in cooperative arrangements such as consortia. Such a 
measure would require users to obtain cost data from external organizations and also determine 
procedures for computing “costs saved” or “costs avoided.” 

Measure: Benchmarks 

Definition: Standards or points of reference in judging quality or value. Participants agreed that 
benchmarking approaches should receive greater attention and detail in the manual. They suggested 
that a section on benchmarking be presented with some sample benchmarks. Users should be 
given some guidance on how to choose institutions against which they should benchmark their own 
institutions. They also need guidance on how to conduct a benchmarking exercise and how to 
determine what types of activities might best be benchmarked. 

Qualitative measures . The group discussed the level of detail needed for conducting the 
various qualitative data collection techniques and generally agreed that in some instances additional 
detail could be provided (e.g. focus groups). But generally, the strategy taken to provide sources 
for additional information was appropriate. Participants made the following suggestions to improve 
existing measures: 
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• Expand the listed topics for focus group discussion to include development, e.g., 
development of alumni. 

• Apprise users of the value in establishing ongoing relationships with individuals 
who participate in focus groups. 

Participants also commented on the need to link qualitative approaches with other data collection 
strategies outlined in the manual. 

User survey . The group spent little time discussing the survey specifically although some 
topics received attention. Participants made the following suggestions to improve the existing user 
survey: 

• Ask survey respondents what they would like to do with the network, not just what 
they are currently doing. 

• Ask survey respondents to place a value on the improvements they would like to 
see in the network. In addition to asking users what they would like to see 
improved or added to the network, ask them what they would be willing to pay for 
those improvements and additions. 

• Anyone planning to administer the survey, or parts of it, should be encouraged to 
find out if there is an office designated for coordinating surveys on campus. Such 
an office may be able to provide timetables for other surveys which would help 
manual users avoid conflicts and comp>etition with other survey administrators. 
Manual users should also investigate opportunities to do cooperative surveying with 
other groups on campus. 

There also was some discussion about the limited faith some participants had in conducting such a 
survey. The belief of some was that the amount of time and effort required for a low response rate, 
and the difficulty of obtaining approval at their campus to conduct a survey, limited the usefulness 
of surveys. 
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Advice to Users of the Manual . Focus group participants suggested that the following 
advice to be given to users of the manual: 

• Users of the manual should be cautioned about the limitations of the measures for 
comparing institutions. There is a need for measures which can be used to compare 
institutions but such a comparison would be difficult or impossible with some of 
the proposed performance measures. 

• Users should be reminded that the disjointed nature of computing and networking 
on many campuses may make application of the performance measures across an 
institution difficult or impossible. 

• Users of the manual should be cautioned that there are privacy issues associated 
with collecting data for these performance measures and that action should be taken 
to protect individuals’ privacy when these measures are used. Users should be 
urged to aggregate all data collected to protect the privacy of individuals. However, 
users should be warned that this protection measure is useful only on central 
servers. 

• Users should be urged to measure networked services or activities which are 
growing or for which there is considerable demand so that they can better plan to 
meet that growth. . 

• Users should be urged to collect data on the proposed measures by different user 
populations within their institutions. 

• Users should be urged to collect data on the proposed measures for different 
projects within their institutions. 

• Users should be reminded that qualitative data may be useful only at a local level. 

• Users should be instructed in ways to combine the data from the various techniques 
outlined in the manual, e.g., quantitative with qualitative, with the user survey. 
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• Users should be provided with advice and guidance on how best to present the 
resulting data to organizational officials and to best support their goals. 

• Users should be urged to use the performance measures for tactical purposes as 
well as for strategic purposes. That is, they should use them to measure their 
success in meeting objectives and to compare their performance from year to year. 

• Users should be given examples of how selected measures can be used to improve 
service, thus providing an incentive to use the measures. 

• Users should understand that the process of performance measurement may be as 
important, itself, as the actual resulting data, i.e., recognizing that ongoing 
evaluation is an important component of network management 

• Users should be encouraged to “get out of their institutional setting” and see/assess 
what others are doing to evaluate networked services. 

Participants also understood that “all the advice in the world still may not help some folks to 
evaluate networked services correctly.” But they did agree that it was important to make the effort 
in the manual. 

Advice to thp Study Team . Participants provided the following general suggestions to the 
study team: 

• Accrediting bodies will be interested in the proposed performance measures, and 
likely will use some of them in accreditation processes, but there are certain 
weaknesses inherent in these measures if they were to be used for accreditation 
processes. Where possible, these weaknesses should be eliminated and where not, 
individuals and institutions should be cautioned about the use of these measures in 
accreditation processes. 

A list of measures used by various accrediting agencies should be appended to the 




manual. 
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• Preface the manual with a statement explaining that it can be used by different kinds 
of institutions and is not seen as a tool to be used exclusively by Carnegie I and II 
research institutions. 

• Consider dropping the word “manual” from the title. “Strategies, guidelines, and 
options” defines the document more accurately and will be a more appropriate 
indicator of its potential use. 

• Consider moving some of the questions from section IV into the user survey. 

• Consider conducting one or two case studies to test the manual at several 
institutions. The results may strengthen the case that evaluation of networking is 
worthwhile. 

• Further specify and make consistent all definitions, especially definitions for the 
following terms: network, academic network, application, resource, service, and 
technology. 

Participants suggested that for the term “network” it may be less important for the 
manual to prescribe such a definition as it is to outline the issues that might help 
users of the manual come up with a definition that would work for them. 

• Develop a definition of the nomadic computing environment. 

• Include a self inventory to help users determine the nature, configuration, and 
extent of networking on the campus. 

• Give a more thorough explanation of the goals of evaluation, which will vary from 
institution to institution. 

• Be consistent with the use of headers and sub-headers throughout the manual as 
current use is oftentimes quite confusing. 

Participants also recognized that it may be difficult to address all of the areas described above. 
Thus, the study team may wish to recognize, in the introduction, the areas that will require 
additional work and testing. 
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In summary, the fcx:us group participants, who are representatives of the group for which 
the performance measures manual is intended, documented the need for and interest in the manual, 
suggested improvements to specific measures and proposed additional measures, and offered 
general suggestions and advice for users of the manual and for the study team. Throughout the 
meeting there was an underlying theme that participants recognized the difficulty in developing 
such measures and procedures. Nonetheless, the sense of the participants was that this effort was 
essential, that we not “get it all correct” the first time, that identifying possible approaches and 
detailing the issues and concerns that have to be addressed in a measurement process, and 
promoting the process for ongoing networking assessment will be a significant contribution. 

Indeed, a number of participants wanted to retain their draft copy of the manual for use 
immediately. 

Following the focus group, two study team members gave a briefing on the project to the 
general membership of CNI. After the briefing, audience members critiqued the project and made 
recommendations for its improvement. Among their recommendations were: 

• reorganize the manual so that the qualitative measures are first, the user survey is 
second, and the quantitative measures are last. The reasoning for this was that 
there already is a great deal of anecdotal information being collected, thus readers of 
the manual will identify most strongly with the qualitative measures and can benefit 
most from learning about methods for the systematic collection of this information 
over time. 

• weed out all prescriptive recommendations and statements 

• emphasize the need to measure longitudinally 

• caution users of the manual that, because technologies and orgamzational structures 
are changing rapidly, these measures will have to be changed often. 

They encouraged the study team members to complete the manual as soon as possible and make it 
available. 
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4.5. Academic Networked Environment Model 

One outcome of this investigation is a model of the academic networked environment (ANE) 
(see Figure 4-1). This model was originally based on a review of the available literature and then 
revised numerous times throughout the study, in response to feedback from the focus groups, 
interviews, discussion, and site visits. 

The proposed model is designed such that many assessment questions of impact can be 
empirically studied. The design has a simple physical nature to facilitate its application yet 
recognizes the complexity of the environment. As in the models previously reviewed, a 
simplification technique of "boxing" or "layering" has been employed. This division is somewhat 
artificial but facilitates understanding. By dividing the network into these logical groupings, one 
can more easily develop empirical tests and quantify the results. This techmque does not reduce the 
value of the model and in fact is the only logical way to evaluate such a complex network. 

The ANE Model has been constructed to facilitate understanding and examination of the 
components of the academic networked enviromnent with the focus on measurement of the impact 
networking has had and will have on the institution. The model is intended to serve as a point of 
reference for discussing and measuring the scope and alignment of academic networked services. 

Although many of the features and components of the ANE model are similar to those in 
models previously mentioned, each model, including ANE has its own specific purpose and 
design. The models, other than ANE, are systems oriented and are not designed with evaluation 
and measurement as key requirements. The ANE model is user oriented, flexible, and has been 
operationalized to facilitate evaluation and measurement 

The ANE model has four major component groups bounded by the perimeter of the 
institution. This perimeter extends beyond the physical boundaries of the institution only to include 
"off campus" connectivity of students and faculty into the institutional network. The component 
groups are described as follows: 




FIGURE 4-1. ANE MODEL 



Academic Networked Environment 




Internet 
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• User Base : Within the ANE model, the user and the user's perspective of the 
networked environment are of major importance. The users, or customers, in this 
environment include teaching and research faculty, students, and administrators, in 
various settings within the institution. Users exist within the environment at all skill 
levels and must be considered the primary reason for the network's existence. The 
users drive the network and its new applications. However, their resistance to 
change may also present impediments to application upgrades. All users share a 
common desire for a system that is easy to use, available when needed, and reliable. 

• Enabling : The ANE model exhibits its uniqueness with emphasis on those 
components of the architecture that "enable" the network's functionality. Very 
basically, these components act as the interface for all other network functions. 

User access to applications, through use of the physical infrastructure, is enabled 
from this area. Enabling services can be classified as campus specific or generic, 
and include distributed computing services, information management services, user 
interface services, and utility services. Providers of these services include Academic 
Computing, Information Management Services, Network Coordinators, and 
Communications Network Providers. Network services are developed, funded, and 
managed from this area. Enabler's often see themselves as utility providers, much 
like the water or power companies, and do not visualize all of the responsibilities 
associated with the enabling function. Enablers need to be both demand and vision 
driven and to meet user expectations at justifiable costs. They must also provide 
user training and help-desk activities. Enabling services must create an environment 
in which new services and applications can be easily introduced and integrated into 
existing services and applications. This component is thus expected to be an 
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especially dynamic one, with technical innovations and new service requests 
continually demanding attention. 

• Technology: The technology component, also known as the physical 
infrastructure, contains the basic processing and communications components of 
the system. The physical connectivity and layout of the network and component 
systems are contained is this area. This pxjrtion of the model is viewed as the 
"backbone" of the architecture. The network speed, bandwidth, and system 
availability and reliability are functions of this area. Ability of the network to 
handle the transmission of voice, data, image, or combinations (multimedia) resides 
in the technology layer. This is an area of rapid transition in both speed and 
physical structure. Proper placement of infrastructure that allows for future 
enhancements at affordable costs is difficult but mandatory. 

• Applications: This area contains all applications or processing tools that run on the 
academic network and actually "do something" for the user. E-mail, data search and 
retrieval programs, interactive education, and on-line registration are examples of 
applications. The quality and quantity of available applications constitute the 
usefulness of this component area. 

The ANE model does have certain limitations. It was designed to facilitate analysis and 
measurement of impacts of the Internet on the institution. It is simplistic in design and categorizes 
functions and participants into manageable components. It is not designed as a techmcal model to 
guide 

The enabling component, which may be thought of in simple terms as the network service 
provision component, is the dominant component and the "glue" that holds the overall network 
together. It is however more than just a service provision piece as this component includes 
plaiming, funding, training, and help>-desk activities in addition to service provision. 
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The technology infrastructure, composed of computing and communicaticais hardware and 
software, provides the backbone of the academic networked environment and the interface to other 
institutions via the Internet. This infrastructure, conceived, implemented, and maintained by the 
enablers, provides significant quantitative measurement opportunities. The extensiveness, and 
efficiency of this physical part of the network have major impact on user satisfaction and network 
use. 

Although users can be categorized for measurement in terms of their position within the 
academic setting and their skill levels, the revised model also includes recognition of user ability to 
drive or provide applications. The interface between users and applications is much more 
significant than originally indicated. To simply measure user satisfaction based on applications 
provided by others is not valid as many applications are now individualized to user appliances. 

This puts additional responsibility into the enabling function to understand these individual 
applications and to provide as much inter-operability as possible between these diverse applications. 

The applications area includes functions that exist within individual workstations, as well as 
those within the campus based public, shared, and research areas. 

The ANE model is designed to facilitate measurement in the areas of extensiveness, 
efficiency, and effectiveness of the various components, leading to assessment of overall impact of 
academic networking on the institution. 

4.6. Assessment Manual 

A major product of this research is the manual. Assessing the Academic Networked 
Environment: Strategies and Options . A copy of the final version of the manual is attached to this 
report and the contents are reviewed briefly here. CNI has distributed the manual to its members at 
no cost and is distributing it to other interested parties for the cost of printing and mailing. 

The manual is to be considered a work in progress. It is not a step-by-step guide to 
network evaluation. Rather, as the titles suggests, it is a collection of strategies and options which 
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institutions may consider in evaluating their networked environments. Some parts of the manual 
will be more relevant for certain institutions that other parts. Any given institution will need to 
select from and modify the measures, instruments, and approaches presented in the manual. 
Additionally there are a number of issues, also reviewed briefly below and presented in more detail 
in the accompanying manual, which must be resolved by individual institutions before they can use 
this manual. The way in which one institution deals with these issues may not be appropriate in 
other institutions. 

The manual includes three sections: a discussion of the collection and use of qualitative data, 
a set of quantitative measures, and an instrument for surveying network users. While any of these 
sections may be used independently of the others, the manual will best be used by integrating the 
various parts. For example, the use of qualitative data may be informative, but the combination of 
qualitative and quantitative data with a user survey will provide a much more complete assessment 
of networking. 

In the qualitative section of the manual, a variety of qualitative techniques for data collection 
and analysis are described. Strengths and weaknesses of each method are presented, and 
references are provided for further information on, and examples of, each technique. Also, a list of 
topics and issues, related to networking, which may be investigated using these qualitative 
techniques, is presented. And finally, strategies for the successful sue of these techniques are 
presented. 

Measures for the assessment of networking are presented in the following broad categories; 
users, costs, network traffic, use, services, and support. For each of the measures in each of these 
areas, the following are provided: definitions of the measures; a discussion of issues which must be 
addressed before that measure can be used; instructions for data collection and analysis; a 
discussion of each measure; and additional suggestions for its use. 

The final part of the mcmual is a survey instrument, accompanied by a discussion of issues 
related to the use of such an instrument eis well eis suggestions for administering the survey. The 
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survey is designed to collect information on the following: user demographics; data on users’ 
experience with computers and their current habits and practices in using their campus network, 
including which resources and services they use, how they use them, and when they use them. 
Users are also asked to evaluate these resources and services as well as the network support 
services available at their institutions. 
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5. CONCLUSIONS & RECOMMENDATIONS 



The purposes of this research were: to define and describe the academic networked 
environment and its component parts; to identify existing measures of network perfonnance and 
impacts; to identify factors which facilitate or inhibit network assessment; and to develop and 
operationalize additional measures. The research questions, presented in the introduction to this 
report, which guided this investigation are reviewed below along with relevant conclusions from 
this investigation and recommendations for future research and practice. 

• What information technologies and services comprise networked information, and 
to what degree are these similar across various academic institutions? 

• Who are the "users" of networked information within the academic setting and how 
might we develop a typology of such users? 

• What are the orgemizational structures used in academic institutions to provide 
networked information services? 

• What are the key factors that appear to affect the overall success of the networked 
envirorunent in an academic setting? 

• What measures can be developed to assess the impacts of networking on the 
academic environment? 

The academic networked environment exists in various configurations which may differ 
from institution to institution. There are some common elements of the networked environment, 
broadly defined in the ANE model presented here as users, applications, an enabling component, 
and an institutional technology infrastructure which cormects to the Internet. In most academic 
institutions, users may be categorized as faculty, students, staff, and others. More specific 
definitions of users will vary from institution to institution. There are some applications which can 
be found in many institutions, including email, OPAC, CWIS, and WWW. But many other 
applications are specific to certain institutions. Likewise, the enabling component and technology 
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infrastructure are likely to be different in different institutions. Thus at a very general level, there is 
a model of the academic networked environment, but at a more specific level, each environment 
may be unique. 

There is currently little assessment of academic networking. In part this is due to the 
variation among networked environments; there is no generic approach or set of measures which 
can be employed universally, nor is there likely to be. Additionally, there are numerous barriers to 
evaluation (e.g., lack of resources, time, training, tools and rewards), the technology is changing 
rapidly, and the environment is increasingly distributed. 

However, there is a need for assessment of networking for purposes of planning, decision 
making, resource allocation, and meeting users’ needs. What’s needed first is development of an 
assessment mindset. That is, university and computing administrators must understand the 
importance of assessment and be willing to assign the necessary resources to the accomplishment 
of the task. Beyond that, useful measures, tools, and procedures must be developed. As this 
research has shown, there may not be a universal set of measures, tools, and procedures. 

However, the strategies and options developed during this investigation, and presented in the 
assessment manual, may provide a starting point for institutions to develop assessment programs. 
Any institution may find certain elements of the manual more appropriate than others and thus 
should pick and choose from the strategies and options presented. Assessment need not be an all- 
or-nothing effort. 

While this study has produced a set of assessment strategies and options and identified 
issues which must be resolved, there is much more work to be done. The investigators believe that 
the next step in this effort should be to field test the manual in a variety of academic settings. That 
is, groups including academic computing and library administrators and staff should attempt to 
implement the measures and approaches developied in this study and the results of their efforts 
should be used to inform the further development of the manual. This will involve a commitment 
of time, effort, and financial resources from institutions which chose to assess their networking. 
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Coding Scheme for Content Analysis of IT Plans 



Benefits 

Learning and teaching 
Future 

Student-faculty communication 
Diversity 
Distance learning 
Learning materials 
Research 

Productivity 
Data access 
Cooperation 
Results communication 
Administration 

Institutional data 
Cost reduction 
Reduction of bureaucracy 
Efficiency 
Decision making 
Cohesion 

Service 

Alumni 
Boundaries 
Information transfer 



Objectives 

Policies and procedures 
IT policies 
Distance learning 
Security 

Development of IT use 

Student purchases 
Curriculum 
Faculty motivation 

Services 

Support 

Training 

Servers 

Community 



Network 

Connection 

Campuses 

Internet 

Interoperability 




Stakeholders 

Funding sources 
Research partners 
K-12 

Resources 

Funding 

Student fee 

Budget 

Cost 

Staffing 

Hardware 

Software 

Supercomputing 

Organizational structure 

Information officer 
Central core 
Decentrahzation 
Planning 
Coordination 

User involvement 

Needs analysis 
Advisory committee 
Surveys 
Liaisons 
User groups 

Evaluation 

Current evaluation 
Necessity of evaluation 
Evaluation methods 
Measures 
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APPENDIX B 



Code Deflnitions and Examples from Content Analysis of IT Plans 



Following is a hierarchical listing of all coding categories used in the content analysis of the 
university computing strategic plans. Definitions are provided for first (x.), second (x.x), and 
thir d (x.x.x) level categories. Examples are listed under the most specific level of each category. 
These examples are sample phrases, taken from the plans, which were coded using those 
categories. The number of plans which include instances of each third level category is also given. 

1. Category Name: Benefits 

Deflnition: Expected positive impacts of implementation of the strategic plan, or of IT in 
general, on the university 

1.1. Category Name: Learning and Teaching 

Deflnition: Improvements to the processes of student learning and faculty teaching 

1.1.1. Category Name: Future (3 plans) 

Definition: Students best prepared for future job and environment 
Example: "When students leave the university, they will be familiar with current 
technology, understand the implications of the information age, and be equipped and ready 
to contribute to their workplace and society" 

1.1.2. Category N ame : Student-Faculty Communication (8 plans) 

Definition: Better communication between students and faculty through 
e-mail 

Example: "'Connectivity' will allow faculty and students to easily communicate 
regarding assignments, grades, special instruction and counseling matters" 

1.1.3. Category Name: Diversity (4 plans) 

Definition: Increased exposure to diversity thanks to the ability of communicating with 
other students on campus and all over the world. 

Example: "Telecommunications permits students to interact electronically with others 
throughout the world. This will be an increasingly important way for... students to learn 
about and accept other peoples and cultures, and to understand the unique role of [our 
state] in the intemationd community" 

1.1.4. Category Name: Distance Learning (7 plans) 

Definition: Increased access to education for people who cannot be on-campus 
traditional students 

Example: "The sophisticated network communications will make it possible for 
successful distance education programs to bring the best of [the university] to anyone, or 
any class, anywhere" 

1.1.5. Category Name: Learning Materials (1 1 plans) 

Definition: "Richer" learning materials thanks to multi-media, simulations, access to 
databases 

Example: "Incorporating many technologies and using them to provide information in a 
variety of ways, such as through simulation exercises, computer-based tutorials, and 
multimedia learning experiences, will provide richness of content" 
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1.2. Category Name: Research 

Definition: Improvements to the faculty research process 

1.2.1. Category Name: Productivity (10 plans) 

Definition: Increased productivity of faculty, without more specifications 
Example: “Enhanced personal productivity, with appropriate information resources and 
computing tools, on campus and elsewhere, that are easy to use and readily available to 
faculty...” 

1.2.2. Category Name: Data Access (12 plans) 

Definition: Increased access to data (bibliographic databases, library catalog, electronic 
journals, data files, grant information) and computing resources 
Example: "Prospecting in archives and repositories of scientific, historical and literary 
data and text will become far easier" 

1.2.3. Category Name: Cooperation (12 plans) 

Definition: Increased cooperation between departments of an institution or between 
researchers at different institutions 

Example: "High potential exists for increasing faculty collaboration throughout [the 
university]... Researchers are able to interact easily with colleagues all over the world who 
may have similar, and perhaps very specialized interest" 

1.2.4. Category Name: Results Communication (5 plans) 

Definition: Improved communication of research results 

Example: "Electronic distribution of articles already has reduced publishing lead times in 
some chsciplines dramatically" 

1.3. Category Name: Administration 

Definition: Improvements to the administrative processes 

1.3.1. Category Name: Institutional Data (13 plans) 

DeDnition: Easy access to institutional data by staff and end-users (schedules, grades, 
financial aid, etc.) 

Example: "The next generation of administrative systems will be developed with 
emphasis on data interchange and ease of use, not only by administrators, but also by 
faculty and students" 

1.3.2. Category Name: Cost Reduction (5 plans) 

Definition: Reduction of costs to perform administrative tasks and processes 
Example: “A productive staff environment will result in cost-effective delivery of quality 
service” 

1.3.3. Category Name: Bureaucracy Reduction (4 plans) 

Definition: Simplification of administrative organizational structure 

Example: "Each administrative office will find it easier to customize its work for the 
ultimate customers, and many intermediary administrative functions will disappear or be 
reduced in importance" 

1.3.4. Category Name: Efficiency (12 plans) 

Deflnition: Increased efficiency of administrative processes 
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Example: "The improved communications that will arise from information technology in 
the future will make our administrative staff more accessible and our administrative 
processes more clear" 

1.3.5. Category Name: Decision-Making (5 plans) 

Definition: Better decision-making thanks to decision support systems and better 
availability of data 

Example: "More in-depth analysis of trend data and other information used to support 
decision making will occur because of increased data availability" 

1.3.6. Category Name: Cohesion (3 plans) 

Definition: More cohesive university 

Example: "Campus wide access to new information services and to an integrated 
administrative data encyclopedia will contribute in important ways to this cohesion" 

1.4. Category Name: Service 

Definition : Improvements in service provided to the larger community 

1.4.1. Category Name: Alumni (3 plans) 

DeDnition: Allowing alumni to keep in touch vrith campus life; provision of services 
(career armouncements, library catalog access etc.) 

Example: "Special bulletin boards for alumni will permit them to keep in touch 
conveniently with former classmates, faculty, and other friends. Network access 
to.. .placement offices(s) will permit dumni to use those placement offices throughout their 
careers" 

1.4.2. Category Name: Boundaries (5 plans) 

DeDnition: Expand the service boundaries of institution into a wider geographical area 
Example: "Faculty, staff, students, alumni/ae, and citizens... will be able to transact all 
necessary business, exchange and obtain information, and work collaboratively..." 

1.4.3. Category Name: Info Transfer (6 plans) 

Definition: Better transfer of information and research results into the community 
Example: "Information technology can provide a convenient and timely method for 
informing the public about all that goes on within the university and for providing them 
with the opportunity to share in the benefits the university makes available within the 
community" 

2. Category Name: Objectives 

Definition: Stated, or implied, objectives of the plans 

2.1. Category Name: Policies and Procedures 
Definition: Development of policies and procedures 

2.1.1. Category Name: IT Policies (14 plans) 

Definition: Development of IT policies and standards 

Example: "Document and communicate telecommunications policies.. ..Clarify policies 
for charging and funding.... Support of interoperability standards for local area networks" 

2.1.2. Category Name: Distance Learning (3 plans) 

Definition: Development of policies for distance learning 
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Example: "Develop policies that address crucial issues relating to instructional 
technology and distance education" 

2.1.3. Category Name: Security (15 plans) 

Definition: Development of policies and procedures to guarantee security and privacy 
of data 

Example: "Define appropriate use, privacy, and security issues related to campus 
communications" 

2.2. Category Name: IT Use 

Definition : Development of IT use in the teaching/leaming process 

2.2.1. Category Name: Student Purchases (9 plans) 

Definition: Incentives for purchase of IT equipment by students 

Example: "Open access sites are needed to serve the large number of students who, for 
various reasons, do not possess their own personal computer. A greater penetration of 
computing technology into the campus can be achieved by encouraging personal 
ownership of microcomputers" 

2.2.2. Category Name: Curriculum (11 plans) 

Definition: Integration of IT into curricula, development of courseware 
Example: "Computing will be integrated into all appropriate curricula, both specific and 
general education, to ensure a level of literacy and numeracy appropriate for a research 
university" 

2.2.3. Category Name: Faculty Motivation (11 plans) 

Definition: Concrete incentives for faculty use of IT in teaching and research 
Example: "Include in the promotion and tenure process recognition and rewards for 
faculty efforts to improve the quality and range of instruction through the use of 
information technologies" 

2.3. Category Name: Services 

Definition: Services provided by computing services units 

2.3.1. Category Name: Support (18 plans) 

Definition: Adequate user support 

Example: "There will be a single, readily identifiable source for help" 

2.3.2. Category Name: Training (15 plans) 

Definition: Adequate user training 

Example: "Increase and appropriately schedule student seminars in the basic computing 
skills most useful to a broad range of disciplines") 



2.3.3. Category Name: Servers ( 17 plans) 

Definition: Development of information servers, such as Campus-wide information 
systems, library catalog, etc. 

Example: "Develop a systemwide information system that provides a wide range of 
university information online, such as job openings, university calendars, course 
schedules, current events, athletic schedules and ticket information, campus bulletins, and 
student and staff directories" 
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2.3.4, Category Name: Community (4 plans) 

Definition: Act as a provider of IT resources and support for the community 
Example: "Support the creation of the community network" 

2.4. Category Name: Network 

Definition: Development of a network infrastructure 

2.4.1. Category Name: Connection (18 plans) 

Deflnition: Connection to the network of on-campus locations. Includes "electronic 
classrooms" 

Example: "The university will have access to a distributed information utility from a 
variety of electronic work environments and from a variety of locations including home, 
dorm room, lab, office, classroom" 

2.4.2. Category Name: Campuses (6 plans) 

Definition: For multi-campuses institutions, adequate network links between campuses 
Example: "Reduce geographic and informatic caste distinctions at the university by 
making resources available to the coordinated campuses" 

2.4.3. Category Name: Internet (15 plans) 

Definition: Adequate access to national and international networks 

Example: "Provide connectivity and appropriate gateways to selected external networks 

which are consistent with the university's missions" 

2.4.4. Category Name: Interoperability (17 plans) 

Definition: Interoperability between different platforms and operating systems, common 
front-end interfaces 

Example: "Access to the distributed information utility will be via a transparent and 
uniform interface" 

3.0. Category Name: Stakeholders 

Definition: Stakeholders other than the ones mentioned in all plans (students, faculty, staff, 
computing services) 

3.1. Category Name: Funding Sources (6 plans) 

Definition: External sources of funding 

Example: " Seek additional opportunities to provide computer access to state agencies, 
federal departments and private activities which will generate additional revenue" 

3.2. Category Name: Research Partners (9 plans) 

Definition: Research projects involving external parties 

Example: "Seek joint venture projects with equipment and software vendors in areas of 
special strength" 

3.3. Category Name: K- 12 (3 plans) 

Definition: Collaboration with elementary and secondary schools 

Example: "Provide the community, especially local high schools, with access to computing 
workshops, seminars, facilities and public information services" 

4.0. Category Name: Resources 
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Definition: Inputs required to provide computing services 

4.1. Category Name: Funding (17 plans) 

DeOnition: Sufficient funding 

Example: "Computing infrastructure requires adequate and predictable funding. 

Investments in computing and information resources should be justified in term of 
educational worth, research support, process improvement and/or administrative 
effectiveness" 

4.2. Category Name: Student Fee (6 plans) 

DeOnition: Necessity of establishing a student technology/computing fee 

Example: "A number of major institutions have used such a fee to help pay for the cost of 

providing an adequate computing enviroiunent to their students" 

4.3. Category Name: Budget (7 plans) 

DeHnition: Budget procures and resource allocations 

Example: "The university.. .will adopt a new model for allocating resources for information 
technology ....The recommended model consists of a mix of 'free' and recharge components" 

4.4. Category Name: Cost (8 plans) 

DeHnition: Actual cost figures and budget requirements 
Example: “Annual hardware maintenance: 10% of initial cost” 

4.5. Category Name: Staffing ( 10 plans) 

DeOnition: Adequate staffing, staff training and development 

Example: "Hire more support persormel, at both the college and system level. ...The 

support staff themselves must be supported with training opportunities" 

4.6. Category Name: Hardware (13 plans) 

Definition: Purchase and maintenance of adequate hardware 

Example: "Assist and support departmental acquisitions of more, newer, faster and more 
powerful workstations and servers" 

4.7. Category Name: Software (14 plans) 

Definition: Purchase and maintenance of adequate software, site licensing 
Example: "Establish an integrated, campus-wide architecture of selected hardware and 
software" 

4.8. Category Name: Supercomputing (12 plans) 

Definition: Access (on-campus or off-campus) to adequate supercomputing resources 
Example: "Provide support and access to national high performance computing centers 
through advanced networks and local support services" 

5. Category Name: Organizational Structure 

Definition: Formal arrangement of the organization 

5.1. Category Name: Information Officer (5 plans) 

Definition: Establishment or existence of a Central information officer at vice presidential 
level 

Example: "The first recommendation is to merger and integrate key technology support 
units and establish a Chief Information Officer (CIO) at the University" 
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5.2. Category Name: Central Core (12 plans) 

Deflnition: Central unit for network, support, guidelines, standards 

Example: “There will be a single, readily identifiable source for help, support, and service 

with networks, computers, and databases” 



5.3. Category Name: Decentralization (13 plans) 

Definition: Decentralization for choice of system, planning, budget, support 
Example: "In keeping with the generally decentralized structure of the university system, 
information technology functions will be both decentralized and centralized. Much of the 
end-user equipment, support, decision-making, and budgeting must occur at the college or 
unit level. There must be central responsibility for activities that require overall consistency 
or for which there are significant economies of scale" 

5.7. Category Name: Planning (9 plans) 

Definition: Establishment or improvement of the planning process 

Example: "Goal six: planning. To integrate information technology perspectives into 

university level planning" 

5.8. Category Name: Coordination (10 plans) 

Definition: Coordination with other units (library, printing, media, etc.) 

Example: "Work closely with the library, university museums, other collections, and 
academic and administrative departments to establish a rich set of online information 
resources and services" 

6.0. Category Name: User Involvement 

Deflnition: Indicators of users’ roles in IT planning, implementation, and support 

6.1. Category Name: Need Analysis (12 plans) 

Deflnition: Decisions based on the analysis of user needs 

Example: "Delivering high quality services demands extraordinary commitment by every 
member of Information Systems and Computing to understanding the needs of end-users" 

6.2. Category Name: Advisory Committee (12 plans) 

Definition: Existence of advisory committees including students, faculty, staff 
Example: "The Faculty Advisory Committees, the Instructional Computing Advisory 
Committee and the Research Computing Advisory Committee will continue to advise the 
Provost and Academic Computing on computing policies and problems" 

6.3. Category Name: Surveys (4 plans) 

Definition: User surveys conducted to assess needs or satisfaction 
Example: "Each organization responsible for the provision of such resources should 
periodically survey a sample of their users and measure their satisfaction and the degree to 
which they believe these several goals have been met" 

6.4. Category Name: Liaisons (5 plans) 

Deflnition: Establishment of a liaison with each department 

Example: "Designate staff liaison for each major academic and administrative unit to be 
responsible for identif 3 dng issues and resolving questions relating to that umt" 




6.5. Category Name: User Groups (3 plans) 

Definition: Establishment of groups of student, staff, and faculty users 

Example: "Encourage the development and growth of formal faculty support groups and 

informal user support groups" 

7.0. Category Name: Evaluation 

Definition: Indicators of attempts to judge the existence, extent, quality, and usefulness of 
IT and the institution 

7.1. Category Name: Current Evaluation (5 plans) 

DeOnition: Analysis of the current strengths and weaknesses of the institution 
Example: “What follows is a straightforward assessment of the university’s current internal 
capacity in key areas of information technology” 

7.2. Category Name: Necessity of Evaluation (5 pl^s) 

DeOnition: Importance of evaluation to process improvement 

Example: "Quality standards must be defined and a quality assurance function formalized to 
ensure that performance is monitored and that services are continually evaluated and 
improved" 

7.3. Category Name: Evaluation Methods (5 plans) 

Definition: Methods of evaluation suggested or already in use. 

Example: “Evaluate continuously new alternatives cind the quality and effectiveness of 
existing information technology” 

7.4. Category Name: Measures (4 plans) 

Definition: Actual measures suggested or already in use 

Example: “Since the introduction of [the conference system] in 1987, a strong computer 
conferencing culture has been established. As of December 31, 1990, there wee 285 
conferences on [the system], with an average of about 4,000 new conference messages per 
month during the last half of 1990” 
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APPENDIX C 

Initial University Focus Group Topics 

1. How would you define and describe the networked environment at your institution? 

2. Is the networked environment you are building, or have built, adequate for your institution? 

3. How do you evaluate the networked environment at your institution? 

4. At the institutional level, how are decisions made regarding the networked environment? 

5. What factors have driven decisions to implement information technologies at your institution? 
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APPENDIX D 



CNI Orlando Focus Group Topics 



Evaluation 



1) How do you know if the information infrastructure you are building, or have built, 
is adequate for your institution? 



What methods and what measures do you use to 
evaluate the infrastructure? 

Did you design it with a certain "quality of service" 
requirement in mind? 

What are the primary obstacles in conducting meaningful 
assessments of infrastructure costs, benefits, impacts, etc.? 

Do you use a benchmark methodology? 

Given the fast pace of information technology change, how 
do you assess the currency of the components of your 
infrastructure? 



2) Have you, or others at your institution, been called upon to provide economic 

justifications for the resources you have assigned to die information infrastructure? 



If so, what measures and indicators have you 
developed to use in your justifications for what has 
been spent or in requesting additional resources? 



3) If called upon by the administration of your school, could you identify and describe 

ways in which the academic networked environment has improved teaching, faculty 
research, student learning, and/or service? 



Are you able to isolate the effects of the networked 
environment from the effects of other variables? 



4) At an institution-wide level, how are decisions made regarding the networked 
environment? 



Planning/Deployment 



Is it a top-down process or are the decisions 
decentralized and driven by the individual 
departments? 
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Who are the key players — within and external to the 
institution? 

How do you coordinate the deployment of 
infrastructure components among the various units 
and levels of the institution? 

What have been the primary obstacles to the success 
of information technologies once they have been 
deployed? 



5) To what extent have the following factors driven the decision to implement 
information technologies at your institution? 

Technological innovation and/or obsolescence 

Competition with other academic institutions 

Policy and/or planning 



Infrastructure Model - Description 

6) In all likelihood, networked environments in academic institutions are similar to 
networked environments in other types of organizations. However, it seems 
equally likely that the nature and mission of Ae university creates certain network 
requirements that are unique. Is it possible to generate a generic model which 
describes and typifies the zicademic networked environment? 

What terms may we use to describe the components? 
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APPENDIX E 



Focus Group Questionnaire 



1 Name: Email Address: 

2. Institution: 

3 . Current Position: 

4. Briefly describe your job responsi bili ties: 

5. How long have you been in your current position? years 

6. How long have you worked in the administration of academic computing services? years 

7. Please list several terms which best describe your area(s) of professional expertise: 



8. Have any major studies or evaluations of network and computing services at your institution been conducted in recent 

years? If so, when? 

9. What are two unique networked services you provide at your i nstitution? 

( 1 ) 

(2)^ 

10. How would you rate the overall quality of your institution's current information infrastructure when compared to other 
schools of the same size and with similar missions? 

Much Worse Slightly Worse Same Slightly Better Much Better 

11. In providing your answer to item #10 above, what do you consider to be the three most important criteria in making this 
assessment? 

( 1 )^ 

( 2 )^ 

( 3 )^ 

12. What do you think are the three most important non-economic challenges facing your institution in the quest to provide 
the best information infrastructure possiWe? 



o 



( 1 ) 

( 2 ) 

( 3 ) 
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